Robotic navigation of robotic surgical systems

ABSTRACT

In certain embodiments, the systems, apparatus, and methods disclosed herein relate to robotic surgical systems with built-in navigation capability for patient position tracking and surgical instrument guidance during a surgical procedure, without the need for a separate navigation system. Robotic based navigation of surgical instruments during surgical procedures allows for easy registration and operative volume identification and tracking. The systems, apparatus, and methods herein allow re-registration, model updates, and operative volumes to be performed intra-operatively with minimal disruption to the surgical workflow. In certain embodiments, navigational assistance can be provided to a surgeon by displaying a surgical instrument’s position relative to a patient’s anatomy. Additionally, by revising pre-operatively defined data such as operative volumes, patient-robot orientation relationships, and anatomical models of the patient, a higher degree of precision and lower risk of complications and serious medical error can be achieved.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional application, which is acontinuation of U.S. Pat. Application Serial No. 16/535,166, which is acontinuation of U.S. Pat. Application Serial No. 15/874,695 filed onJan. 18, 2018, which claims priority to provisional application SerialNo. 62/447,884 filed on Jan. 18, 2017, which is incorporated in itsentirety herein.

FIELD OF INVENTION

This invention relates generally to robotic surgical systems. Forexample, in certain embodiments, the invention relates to roboticsurgical systems with built-in navigation capability for positiontracking during a surgical procedure.

BACKGROUND

Many surgeries (e.g., spinal and orthopedic surgeries) currently requirethe use of medical images displayed to the surgeon in real time, toprovide visual navigation to support surgical action, gestures, anddecision making. Using medical images, a surgeon can be provided withreal-time feedback of the position of surgical instruments in referenceto patient anatomy (as pictured by medical images).

Current surgical navigation systems are based on the principle oftracking. For example, a navigation system generally contains a trackingdevice which measures position of the surgical instruments and patientin real time. Different tracking devices operate on differentprinciples. The most popular are optical tracking and electro-magnetictracking. Optical tracking uses camera systems that measure fiducials(e.g., reflective spheres, LEDs) configured on markers having definedand known anatomy. In this way, the position and orientation of a markercan be determined and, thus, the position and orientation of the elementto which they are affixed (e.g., surgical instruments, patient anatomy)can be tracked as well. In electro-magnetic tracking, the camera of anoptical tracking system is replaced by a field generator. Markers aresensor units (e.g., coils) which measure spatial changes in thegenerated field. In this way, the position and orientation of the EMmarker can be determined in reference to field generator.

There are commercial navigation systems available on the market, forexample, Stealthstation S7 from Medtronic, Curve from Brain lab,electro-magnetic Kick EM from Brainlab and others. A typical workflowfor use of these navigation systems follows the steps of: obtainingpatient images, fixing a reference on the patient, registering thepatient, and tracking instrument and patients to show real-time feedbackto the surgeon. Patient images may be generated by CT, MRI, orflat-panel fluoroscopy (e.g., 0-Arm), for example. References fixed tothe patient include optical markers with a fiducial mark orelectro-magnetic markers. Markers are fixed using, for example, bonescrews or bone fixations. Registering the patient requires defining arelationship between the patient images and the fixed marker.Registration may be performed using a point-to-point method, surfacematching, or automatic registration based on images taken with fixedmarkers (e.g., on the patient’s anatomy).

Current navigation systems have numerous limitations. These navigationsystems are generally difficult to use and require additional surgeonand/or staff training for their operation. The navigation systems takeup a lot of space in the operating room. For example, precious realestate in the operating room space may be occupied by a stand-alonenavigation station/console with tracking camera, screens used for visualfeedback, cords and plugs, power systems, controllers, and the like,creating additional clutter. Also, current optical navigation systemshave a line of sight requirement, in that all tracked instruments mustremain visible to the camera in order to be tracked. If there are notenough fiducials (e.g., spheres, LEDs) visible marker positions may notbe able to be determined. An additional risk is that fiducial positioncan be misread by the navigation system due to obfuscation (e.g., by adrop of blood or transparent drape). Electro-magnetic navigation systemshave problems with metal and ferromagnetic materials placed in fieldwhich can influence thefield and thus add error to marker positionmeasurement. Moreover, navigation systems are expensive, costingapproximately $200k or more. The precision of the measurement isrelatively low in commercial stations, for example, on the level of O.3mm RMS error for position measurement. Additionally, the measurementsare noisy. The frequency of measurement is low (i.e., approximately 20Hz).

The most severe limitation of known navigation systems is that thenavigation desynchronizes over time. The surgeon registers the patientinitially at the beginning of the surgical procedure, using one or moremarkers attached to the patient’s anatomy. Throughout the surgicalprocedure, the patient’s anatomy shifts due to movement of the patientor as a result of the surgical procedure itself. For example, insurgeries involving elongation steps or realignment steps, the patient’sanatomy will have a different position and orientation relative to thefiducial marker(s) after the elongation or realignment. Only the arealocal to the fiducial marker(s) remains accurate to the physical realityof the patient’s anatomy. The error between the reality of the patient’sanatomy and the assumed reality based on the initial registrationincreases with distance from the fiducial marker(s). Thus, in manysurgical procedures being performed today using robotic surgical systemswith known navigation systems, as the procedure progresses, thenavigation system becomes more desynchronized and thus less useful tothe surgeon, as it is less reflective of real life. Likewise, thelikelihood of complications and serious medical error increases.

There are robotic surgical systems which are combined with a navigationsystem, for example, Excelsius GPS from Globus, ROSA SPINE from Medtech(currently Zimmer/Biomet), MAKO from Stryker and others. However, thesesystems use the same navigation system approach as the aforementionedknown systems. The use of a robotic arm aids a surgeon in making precisegestures, but the systems inherit the disadvantages of navigationsystems especially: training requirements, required space, line ofsight, price and low precision of optical navigation over the course ofa surgical procedure. The likelihood of complications and seriousmedical errors due to desynchronization are not reduced in these roboticsurgical systems.

Thus, there is a need for robotic surgical systems for instrumentguidance and navigation wherein the patient registration can be updatedovertime to accurately reflect the instant patient situation.

SUMMARY

In certain embodiments, the systems, apparatus, and methods disclosedherein relate to robotic surgical systems with built-in navigationcapability for patient position tracking and surgical instrumentguidance during a surgical procedure, without the need for a separatenavigation system. Robotic based navigation of surgical instrumentsduring surgical procedures allows for easy registration and operativevolume identification and tracking. The systems, apparatus, and methodsherein allow re-registration, model updates, and operative volumes to beperformed intra- operatively with minimal disruption to the surgicalworkflow. In certain embodiments, navigational assistance can beprovided to a surgeon by displaying a surgical instrument’s positionrelative to a patient’s anatomy. Additionally, by revisingpre-operatively defined data such as operative volumes, patient-robotorientation relationships, and anatomical models of the patient, ahigher degree of precision and lower risk of complications and seriousmedical error can be achieved.

In certain embodiments, described herein is a robotic surgical systemcomprising a robotic arm that has a directly or indirectly attachedforce sensor that is used to collect spatial coordinates of a patient’sanatomy. A surgeon can maneuver the robotic arm between different pointsin space and contact the patient at different points on the patient’sanatomy with an instrument attached to the robotic arm. In certainembodiments, the instrument comprises the force sensor. Contact isdetermined based on haptic feedback registered by the force sensor. Incertain embodiments, a threshold (e.g., a magnitude of haptic feedback)must be exceeded in order to register the contact as belonging to thepatient’s anatomy. Furthermore, in this way, the magnitude of hapticfeedback can be used to determine the type of tissue being contacted(e.g., because bone is harder than soft tissue). In some embodiments,the instrwnent contacts a specially engineered fiducial marker attachedto the patient’s anatomy at one or more of a set of orienting contactpoints (e.g., indents), wherein the fiducial marker has an establishedspatial relationship with the patient’s anatomy (e.g., given its knownsize and intended attachment at a specific known location on thepatient’s anatomy). A plurality of spatial coordinates can be recordedand stored electronically using a plurality of contacts of theinstrwnent to the patient’s anatomy.

A set of spatial coordinates recorded from contact of the instrwnentwith the patient’s anatomy can be used to perform many navigational andsurgical guidance functions such as registration, modeling volumeremoval, re-registration, defining operational volumes, revisingoperational volumes after re-registration, converting stored volumemodels to physical locations, and displaying surgical relative to apatient’s anatomy on navigation screens.

By mapping surfaces defined by sets of coordinates obtained bycontacting a patient’s anatomy to a model of their anatomy (e.g., frommedical image data), a coordinate mapping can be recorded thattranslates between the coordinate systems of the model and physicalreality. For example, the model can be represented in a medical imagedata coordinate system and physical reality in a robot coordinatesystem. Thus, a robotic surgical system can know the physical locationof the patient’s anatomy relative to a surgical instrument attachedhereto. Using the combination of haptic-feedback-generated sets ofspatial coordinates and sets of medical image data coordinates thatmodel the surface of a patient’s anatomy, the aforementionednavigational and surgical guidance functions can be performed quicklyand with high precision pre-and/or intra-operatively.

A set of spatial coordinates can be used to register the patient’sanatomy with a model of the patient’s anatomy derived from medical imagedata. Medical image data may be used from any relevant technique.Examples include, but are not limited to, x-ray data, tomographic data(e.g., CT data), magnetic resonance imaging (MRI) data, and flat-panelfluoroscopy (e.g., 0-Arm) data. In some embodiments, such medical imagedata is taken intra- operatively. In this way, the physical position andorientation of a patient’s anatomy can be mapped to the model of thepatient’s anatomy and the robotic surgical system can know where it isin relation to the anatomy at all times.

A set of spatial coordinates can be used to update a model of apatient’s anatomy after volume removal by determining the volume removedusing additional contacts between the instrument and the patient’sanatomy. Contacts determined to be made on the new surface of theanatomy that correspond to coordinates inside the volume of the modelcan be used to update the surface of the model to reflect the patient’snew anatomy.

A set of spatial coordinates collected intra-operatively can be used tore-register the patient’s anatomy. In this way, changes that haveoccurred in the anatomy, such as re- orientation or repositioning of allof or a part of the anatomy, can be used to update the mapping betweenthe robot coordinate system and the medical image data coordinate systemas well as the model of the patient’s anatomy.

A set of spatial coordinates can be used to define an operationalvolume, wherein the movement of a surgical instrument is constrained tobe within the operational volume during a part of a procedure. Forexample, this can be used to limit the volume of bone removed by asurgeon. The operational volume may be defined by contacting points onthe patient’s anatomy and using those as vertices of a surface or bymapping a model of the anatomical volume to the set of spatialcoordinates and then defining the operational volume as the mappedanatomical model expressed in the robotic surgical system’s coordinatesystem. The operational volume can be updated after a re- registrationto accurately reflect the patient’s current anatomy and/or anatomicalposition and orientation. Likewise, a stored model (e.g., a modelgenerated from medical image data) can be used to define a physicallocation of a portion (orentirety) of that model by using a coordinatemapping.

A rendering of the patient’s anatomy and a surgical instrument’sposition relative to the anatomy can be displayed on a navigation screenfor use and/or reference by a surgeon using the methods and systemsdescribed herein. By using a coordinate mapping, the location of aterminal point of a surgical instrument can be displayed along with arendering of the patient’s anatomy such that a surgeon can observe anaccurate representation of the space or distances between the terminalpoint and the patient’s anatomy. This can be used to visualizetrajectories, positions, and orientations of the surgical instrumentrelative to the patient’s anatomy. A surgeon may use this to monitor theprogress of a surgical procedure, avoid a serious medical error, and/orimprove patient outcome by revising the planned surgical procedure. Forexample, the surgeon can use the navigational display in deciding toalter the volume planned for removal or the planned orientation ortrajectory of the surgical tool when removing the volume. This can bedone intra-operatively.

In another aspect, the disclosed technology includes a robot-basednavigation system for real-time, dynamic re-registration of a patientposition (e.g., position of vertebrae of a patient) during a procedure(e.g., surgical procedure, e.g., a spinal surgery) (e.g., a combinednavigation/robotic system), the system including: (a) a robotic arm(e.g., having 3, 4, 5, 6, or 7 degrees of freedom) including: an endeffector [e.g., said end effector including a surgical instrument holderfor insertion or attachment of a surgical instrument therein/thereto,e.g., said robotic arm designed to allow direct manipulation of saidsurgical instrument by an operator (e.g., by a surgeon) when thesurgical instrument is inserted in/attached to the surgical instrumentholder of the end effector, said manipulation subject to hapticconstraints based on the position of the end effector (and/or thesurgical instrument) in relation to the patient, e.g., said surgicalinstrument having known geometry and fixed position in relation to thesurgical instrument holder]; (ii) a position sensor for dynamicallytracking a position of the end effector [e.g., during a surgicalprocedure) (and/or for dynamically tracking one or more points of thesurgical instrument, e.g., in 30 space, e.g., during a surgicalprocedure) (e.g., at a rate of at least 100 Hz, e.g., 250 Hz or greater,e.g., 500 Hz or greater, e.g., 1000 Hz or greater (positiondeterminations per second)]; and (iii) a force feedback subsystem (e.g.,including sensor(s), actuator(s), controller(s), servo(s), and/or othermechanisms) for delivering a haptic force to a user manipulating the endeffector (e.g., manipulating a surgical instrument inserted in theinstrument holder of the end effector) (e.g., wherein the force feedbacksubsystem includes one or more sensors for performing one or more of (I)to (IV) as follows: (I) detecting a resistive force caused by thesurgical instrument contacting, moving against, penetrating, and/ormoving within a tissue of the patient, (II) distinguishing betweencontacted tissue types (e.g., determining when contacted tissue meets orexceeds a threshold resistance, e.g., when the tissue is bone), (III)detecting a force delivered by the operator (e.g., the surgeon, e.g.,delivered by direct manipulation of the surgical instrument inserted inthe surgical instrument holder of the end effector) (e.g., to causemovement of the surgical instrument and, therefore, the end effector),and (IV) distinguishing between the force delivered by the operator andthe resistive force caused by movement of the surgical instrument inrelation to the tissue of the patient; (b) a display [e.g., attached to,embedded within, or otherwise positioned in relation to the robotic armbeing directly manipulated by the operator (e.g., surgeon) to allow forunimpeded visual feedback to the operator during the procedure, e.g.,wherein the display is positioned beneath a transparent orsemitransparent sterile drape, e.g., wherein the display has touchsensors for control of the display during use]; and (c) a processor of acomputing device programmed to execute a set of instructions to: (i)access (e.g., and graphically render on the display) an initialregistration of the patient position (e.g., position of the vertebrae ofthe patient) (e.g., via medical images of the patient, e.g., MRI, CT,X-rays, SPECT, ultrasound, or the like, e.g., said medical imagesobtained pre-operatively)(e.g., for storing and/or rendering a 3Drepresentation, e.g., a 3D graphical representation and/or a 3D hapticrepresentation, of an initial patient situation, e.g., wherein the 3Dgraphical representation is the same as or different from the 3D hapticrepresentation, e.g., for use in displaying a real-time graphicalrepresentation of the patient situation (e.g., a target anatomy) on thedisplay and/or for use in dynamically determining a force feedbackdelivered to the operator, e.g., during a surgical procedure, via theforce feedback subsystem); (ii) dynamically determine a position of theend effector (e.g., dynamically determine a 3D position of one or morepoints of a surgical instrument positioned in relation to the endeffector, e.g., within an instrument holder of the end effector); (iii)dynamically determine a force received by the end effector and/or aforce to be delivered to the end effector [e.g., a force received byand/or a force to be delivered to the end effector via the surgicalinstrument, e.g., dynamically perform one or more of (I) to (IV) asfollows: (I) determine a resistive force caused by the surgicalinstrwnent contacting, moving against, penetrating, and/or moving withina tissue of the patient, (II) distinguish between contacted tissue types(e.g., determining when contacted tissue meets or exceeds a thresholdresistance, e.g., when the tissue is bone), (III) detect a forcedelivered by the operator (e.g., the surgeon, e.g., delivered by directmanipulation of the surgical instrument inserted in the surgicalinstrument holder of the end effector) (e.g., to cause movement of thesurgical instrwnent and, therefore, the end effector), and (IV)distinguish between the force delivered by the operator and theresistive force caused by movement of the surgical instrument inrelation to the tissue of the patient, (e.g., using the force feedbacksubsystem, e.g., the force to be dynamically determined at a rate of atleast 100 Hz, e.g., 250 Hz or greater, e.g., 500 Hz or greater, e.g.,1000 Hz or greater)]; (iv) dynamically determine a position of theposition sensor of the robotic arm for dynamically tracking the endeffector (e.g., determine a position of the position sensor of therobotic arm upon contact of the surgical instrument with bone tissue ofthe patient, or other target tissue of the patient) (e.g., dynamicallyupdate the recorded position of the position sensor at a rate of atleast 100 Hz, e.g., 250 Hz or greater, e.g., 500 Hz or greater, e.g.,1000 Hz or greater); (v) dynamically re-register the patient positionbased at least in part on an updated position of the end effectordetermined by the position sensor [(e.g., during a surgical procedure)(e.g., update the 30 representation of the patient situation, e.g., the30 graphical representation and/or the 30 haptic representation, basedat least in part on the updated position of the end effector when it isdetermined (e.g., via the force feedback subsystem) that the surgicalinstrument is in contact with a target anatomy, e.g., in contact withbone of the patient) (e.g., using a surface matching algorithm keyed tothe initial (or previous) registration) (e.g., dynamically re-registerthe patient position upon detected contact of the end effector, or thesurgical instrument, or a portion or component of the surgicalinstrument or end effector, with a pre-planned fiducial (e.g., amechanical marker, e.g., a marker fixed to the patient, e.g., attachedto target anatomy, e.g., attached to a vertebra)) (e.g., dynamicallyre-register the patient position upon detected proximity of the endeffector, or the surgical instrument, or a portion or component of thesurgical instrument or end effector, with a pre-planned fiducial (e.g.,a mechanical marker, e.g., a marker fixed to the patient, e.g., attachedto target anatomy, e.g., attached to a vertebra)) (e.g., dynamicallyre-register the patient position based upon the updated position of theend effector determined upon operator command, e.g., surgeon pressing abutton or otherwise activating a graphical or tactile user interfacewhen a re- registered representation is desired)]; (vi) graphicallyrender the re-registered patient position for viewing on the display(e.g., graphically render the updated 3D graphical representation); and(vii) dynamically determine a force feedback to deliver via the forcefeedback subsystem (e.g., to an operator of the robotic arm during thesurgical procedure) based at least in part on the re- registered patientposition [(e.g., based at least on the updated 3D representation of thepatient situation and a current position of the end effector (and/or thesurgical instrument) (e.g., subject to predetermined go/no-go zones)(e.g., thereby permitting, facilitating, directing (e.g., imposing ahaptic detent or well), inhibiting (e.g., imposing a speed constraint),and/or disallowing movement of the surgical instrument in go/no-gozones, e.g., by direct manipulation of the surgical instrument by theoperator, e.g., surgeon)].

In another aspect, the disclosed technology includes a method ofregistering a patient’s anatomy with an instrument attached to anend-effector of a robotic arm of a robotic surgical system, the methodincluding the steps of: receiving, by a processor of a computing device,haptic feedback, from a force sensor attached directly or indirectly tothe robotic arm, prompted by movement of the end-effector (e.g., towardsa patient); determining, by the processor, that the haptic feedbackcorresponds to contact of the instrument with a material (e.g., having acertain density or certain mechanical properties) (e.g., based at leaston a magnitude of the haptic feedback exceeding a threshold) (e.g.,additionally based on the location of at least one point on theinstrument) (e.g., wherein the material is bone); determining, by theprocessor, a set of spatial coordinates, wherein the set of spatialcoordinates includes a spatial coordinate for each contact of theinstrument with the material expressed using a robot coordinate system,(e.g., relative to the position of the end- effector), wherein eachspatial coordinate corresponds to a point on the surface of ananatomical volume (e.g., a point on a surface of a bone); receiving, bythe processor, a set of medical image data coordinates expressed using amedical image data coordinate system that correspond to a patientanatomy surface (e.g., determined from tomographic patient data (e.g.,CT data, MRI data)); mapping, by the processor, (e.g., using surfacematching) the surface corresponding to the set of spatial coordinates tothe patient anatomy surface corresponding to the set of medical imagedata coordinates (e.g., by generating a transformation array ortransformation matrix); generating, by the processor, a coordinatemapping between the robot coordinate system and the medical image datacoordinate system based on the mapping between the surface correspondingto the set of spatial coordinates and the surface corresponding to theset of medical image data coordinates; and storing, by the processor,the coordinate mapping, thereby registering the patient’s anatomy (e.g.,for navigational use by a surgeon during a surgical procedure).

In certain embodiments, the method includes the step of: outputting, bythe processor rendering data for display (e.g., on a display of therobotic surgical system; e.g., on a display on the robotic arm), whereinthe rendering data corresponds to a representation of a position of amember and at least a portion of the medical image databased on thecoordinate mapping, wherein the member is selected from the groupconsisting of: the end- effector, the instrument, and a surgicalinstrument.

In certain embodiments, the method includes the steps of: generating, bythe processor, new rendering data by modifying the rendering data basedon a change in the end- effector’s position; and outputting, by theprocessor, the new rendering data for display.

In certain embodiments, a fiducial marker includes the material (e.g.,the end-effector contacts a fiducial marker with known size and shapesuch that the spatial coordinate is determined using a spatialrelationship between the fiducial marker and the patient’s anatomy).

In certain embodiments, the robotic arm is active and non-back drivable.

In certain embodiments, the robotic surgical system includes theprocessor.

In certain embodiments, the method includes storing, by the processor, apatient anatomy model wherein the patient anatomy model is defined bythe patient anatomy surface expressed in the robot coordinate system.

In one aspect, the disclosed technology includes a robotic surgicalsystem for registering a patient’s anatomy with an instrument attachedto an end-effector of a robotic arm of the robotic surgical system, thesystem including: a robotic arm with amend-effector having an instrumentattached thereto; a force sensor attached directly or indirectly to therobotic arm (e.g., the force sensor located between the instrument andthe robotic arm); and a processor and a memory having instructionsstored thereon, wherein the instructions, when executed by theprocessor, cause the processor to: receive haptic feedback, from theforce sensor, prompted by movement of the end-effector (e.g., towards apatient); determine that the haptic feedback corresponds to contact ofthe instrument with a material (e.g., having a certain density orcertain mechanical properties) (e.g., based at least on a magnitude ofthe haptic feedback exceeding a threshold) (e.g., additionally based onthe location of at least one point on the instrument) (e.g., wherein thematerial is bone); determine a set of spatial coordinates, wherein theset of spatial coordinates includes a spatial coordinate for eachcontact of the instrument with the material, expressed using a robotcoordinate system, (e.g., relative to the position of the end-effector),wherein each spatial coordinate corresponds to a point on the surface ofan anatomical volume (e.g., a point on a surface of a bone); determine aset of medical image data coordinates expressed using a medical imagedata coordinate system that correspond to a patient anatomy surface(e.g., determined from tomographic patient data (e.g., CT data, MRIdata)); map (e.g., using surface matching) the surface corresponding tothe set of spatial coordinates to the patient anatomy surfacecorresponding to the set of medical image data coordinates (e.g., bygenerating a transformation array or transformation matrix); generate acoordinate mapping between the robot coordinate system and the medicalimage data coordinate system based on the mapping between the surfacecorresponding to the set of spatial coordinates and the surfacecorresponding to the set of medical image data coordinates; and storethe coordinate mapping, thereby registering the patient’s anatomy (e.g.,for navigational use by a surgeon during a surgical procedure).

In certain embodiments, the instructions, when executed by theprocessor, cause the processor to: output rendering data for display(e.g., on a display of the robotic surgical system; e.g., on a displayon the robotic arm), wherein the rendering data corresponds to arepresentation of a position of a member and at least a portion of themedical image data based on the coordinate mapping, wherein the memberis selected from the group consisting of: the end-effector, theinstrument, and a surgical instrument.

In certain embodiments, the instructions, when executed by theprocessor, cause the processor to: generate new rendering data bymodifying the rendering data based on a change in the end-effector’sposition; and output the new rendering data for display.

In certain embodiments, a fiducial marker includes the material (e.g.,the end- effector contacts a fiducial marker with known size and shapesuch that the spatial coordinate is determined using a spatialrelationship between the fiducial marker and the patient’s anatomy).

In certain embodiments, the robotic arm is active and non-back drivable.

In certain embodiments, the robotic surgical system includes theprocessor.

In certain embodiments, the instructions, when executed by theprocessor, cause the processor to: store a patient anatomy model whereinthe patient anatomy model is defined by the patient anatomy surfaceexpressed in the robot coordinate system.

In one aspect, the disclosed technology includes a method of updating amodel of a patient’s anatomy after volume removal with an instrumentattached to an end-effector of a robotic arm of a robotic surgicalsystem, the method including the steps of: receiving, by a processor ofa computing device, haptic feedback, from a force sensor attacheddirectly or indirectly to therobotic arm, prompted by movement of theend-effector (e.g., towards a patient); determining, by the processor,that the haptic feedback corresponds to contact of the instrument with amaterial (e.g., having a certain density or certain mechanicalproperties) (e.g., based at least on a magnitude of the haptic feedbackexceeding a threshold) (e.g., additionally based on the location of atleast one point on the instrument) (e.g., wherein the material is bone);determining, by the processor, a set of spatial coordinates, wherein theset of spatial coordinates includes a spatial coordinate for eachcontact of the instrument with the material, expressed using a robotcoordinate system (e.g., relative to the position of the end-effector),wherein each spatial coordinate corresponds to a point on the surface ofan anatomical volume (e.g., a point on a surface of a bone); receiving,by the processor, a set of medical image data coordinates thatcorrespond to the surface of a volume of the patient’s anatomy, whereineach medical image data coordinate in the set of medical image datacoordinates is expressed using a medical image data coordinate system;receiving, by the processor, a coordinate mapping between the robotcoordinate system and the medical image data coordinate system (e.g., atransformation array or transformation matrix); determining, by theprocessor, one or more interior spatial coordinates in the set ofspatial coordinates that correspond to points inside the surface of thevolume of the patient’s anatomy based on the set of medical image datacoordinates and the coordinate mapping; generating, by the processor, aset of interior medical image data coordinates, wherein the set ofinterior medical image data coordinates includes an interior medicalimage data coordinate for each of the one or more interior spatialcoordinates using the coordinate mapping; modifying, by the processor,the set of medical image data coordinates that define the surface of thevolume of the patient’s anatomy with the set of interior medical imagedata coordinates such that a first volume defined by the set of medicalimage data coordinates is larger than a second volume defined by themodified set of medical image data coordinates; and storing, by theprocessor, the modified set of medical image data coordinates (e.g., fordisplaying to a surgeon), thereby updating the model of the patient’sanatomy.

Inone aspect, the disclosed technology includes a system for updating amodel of a patient’s anatomy after volume removal with an instrumentattached to an end-effector of a robotic arm of a robotic surgicalsystem, the system including: a robotic arm with an end-effector havingan instrument attached thereto; a force sensor attached directly orindirectly to the robotic arm (e.g., the force sensor located betweenthe instrument and the robotic arm); and a processor and a memory havinginstructions stored thereon, wherein the instructions, when executed bythe processor, cause the processor to: receive haptic feedback, from theforce sensor, prompted by movement of the end- effector (e.g., towards apatient); determine that the haptic feedback corresponds to contact ofthe instrument with a material (e.g., having a certain density orcertain mechanical properties) (e.g., based at least on a magnitude ofthe haptic feedback exceeding a threshold) (e.g., additionally based onthe location of at least one point on the instrument) (e.g., wherein thematerial is bone); determine a set of spatial coordinates, wherein theset of spatial coordinates includes a spatial coordinate for eachcontact of the instrument with the material, expressed using a robotcoordinate system (e.g., relative to the position of the end-effector),wherein each spatial coordinate corresponds to a point on the surface ofan anatomical volume (e.g., a point on a surface of a bone); receive aset of medical image data coordinates that correspond to the surface ofa volume of the patient’s anatomy, wherein each medical image datacoordinate in the set of medical image data coordinates is expressedusing a medical image data coordinate system; receive a coordinatemapping between the robot coordinate system and the medical image datacoordinate system (e.g., a transformation array or transformationmatrix);determine one or more interior spatial coordinates in the set ofspatial coordinates that correspond to points inside the surface of thevolume of the patient’s anatomy based on the set of medical image datacoordinates and the coordinate mapping; determine a portion of thevolume of the patient’s anatomy that has been removed using the one ormore interior spatial coordinates; generate a set of interior medicalimage data coordinates, wherein the set of interior medical image datacoordinates includes an interior medical image data coordinate for eachof the one or more interior spatial coordinates using the coordinatemapping; modify the set of medical image data coordinates that definethe surface of the volante of the patient’s anatomy with the set ofinterior medical image data coordinates such that a first volume definedby the set of medical image data coordinates is larger than a secondvolume defined by the modified set of medical image data coordinates;and store the modified set of medical image data coordinates (e.g., fordisplaying to a surgeon), thereby updating the model of the patient’sanatomy.

In one aspect, the disclosed technology includes a method ofre-registration patient’s anatomy during a surgical procedure with aninstrument attached to an end-effector of a robotic arm of a roboticsurgical system, the method including the steps of: receiving, by aprocessor of a computing device, haptic feedback, from a force sensorattached directly or indirectly to the robotic arm, prompted by movementof the end-effector (e.g., towards a patient); determining, by theprocessor, that the haptic feedback corresponds to contact of theinstrument with a material (e.g., having a certain density or certainmechanical properties) (e.g., based at least on a magnitude of thehaptic feedback exceeding a threshold) (e.g., additionally based on thelocation of at least one point on the instrument) (e.g., wherein thematerial is bone); determining, by the processor, a set of spatialcoordinates, wherein the set of spatial coordinates includes a spatialcoordinate for each contact of the instrument with the material,expressed using the robot coordinate system (e.g., relative to theposition of the end-effector), wherein each spatial coordinatecorresponds to a point on the surface of an anatomical volume (e.g., apoint on a surface of a bone); receiving, by the processor, a coordinatemapping between a robot coordinate system and a medical image datacoordinate system (e.g., a transformation array or transformationmatrix), wherein the robot coordinate system corresponds to a physicalcoordinate system of the end-effector; updating, by the processor, thecoordinate mapping based on a mapping of the surface corresponding tothe set of spatial coordinates; and storing, by the processor, theupdated coordinate mapping (e.g., to provide an accurate navigationalmodel for use by a surgeon during a surgical procedure), thereby re-registering the patient’s anatomy, the mapping is generated usingsurface matching.

In certain embodiments, the updating step includes: determining, by theprocessor, a set of modeling coordinates, by converting, using thecoordinate mapping, a set of medical image modeling coordinates definingthe surface of a volume of a patient anatomy, wherein the set ofmodeling coordinates are expressed in the robot coordinate system anddefine an anticipated location of the surface of the volume, and the setof medical image modeling coordinates have been generated from medicalimaging data; and mapping, by the processor, (e.g., using surfacematching,) the surface corresponding to the set of spatial coordinatesto the patient anatomy surface corresponding to the set of modelingcoordinates (e.g., by generating a transformation array ortransformation matrix); and updating, by the processor, the coordinatemapping based on the mapping of the surface corresponding to the set ofspatial coordinates to the set of modeling coordinates.

In certain embodiments, the updating step includes: receiving, by theprocessor, a set of modeling coordinates, wherein the set of modelingcoordinates are expressed in the robot coordinate system and define thesurface of a volume of a patient anatomy; mapping, by the processor,(e.g., using surface matching,) the surface corresponding to the set ofspatial coordinates to the patient anatomy surface corresponding to theset of modeling coordinates (e.g., by generating a transformation arrayor transformation matrix); and updating, by the processor, thecoordinate mapping based on the mapping of the surface corresponding tothe set of spatial coordinates to the set of modeling coordinates.

In one aspect, the disclosed technology includes a system forre-registering a patient’s anatomy during a surgical procedure with aninstrument attached to an end-effector of a robotic arm of a roboticsurgical system, the system including: a robotic arm with anend-effector having an instrument attached thereto; a force sensorattached directly or indirectly to the robotic arm (e.g., the forcesensor located between the instrument and the robotic arm); and aprocessor and a memory having instructions stored thereon, wherein theinstructions, when executed by the processor, cause the processor to:receive haptic feedback, from the force sensor, prompted by movement ofthe end-effector (e.g., towards a patient); determine that the hapticfeedback corresponds to contact of the instrument with a material (e.g.,having a certain density or certain mechanical properties) (e.g., basedat least on a magnitude of the haptic feedback exceeding a threshold)(e.g., additionally based on the location of at least one point on theinstrument) (e.g., wherein the material is bone); determine a set ofspatial coordinates, wherein the set of spatial coordinates includes aspatial coordinate for each contact of the instrument with the material,expressed using the robot coordinate system (e.g., relative to theposition of the end-effector), wherein each spatial coordinatecorresponds to a point on the surface of an anatomical volume (e.g., apoint on a surface of a bone); receive a coordinate mapping between arobot coordinate system and a medical image data coordinate system(e.g., a transformation array or transformation matrix), wherein therobot coordinate system corresponds to a physical coordinate system ofthe end-effector; update the coordinate mapping based on a mapping ofthe surface corresponding to the set of spatial coordinates; and storethe updated coordinate mapping (e.g., to provide an accuratenavigational model for use by a surgeon during a surgical procedure),thereby re-registering the patient’s anatomy.

In certain embodiments, the mapping is generated using surface matching.

In certain embodiments, the updating step includes instructions that,when executed by the processor, cause the processor to: determine a setof modeling coordinates, by converting, using the coordinate mapping, aset of medical image modeling coordinates defining the surface of avolume of a patient anatomy, wherein: the set of modeling coordinatesare expressed in the robot coordinate system and define an anticipatedlocation of the surface of the volume, and the set of medical imagemodeling coordinates have been generated from medical imaging data; andmap (e.g., using surface matching,) the surface corresponding to the setof spatial coordinates to the patient anatomy surface corresponding tothe set of modeling coordinates (e.g., by generating a transformationarray or transformation matrix); and update the coordinate mapping basedon the mapping of the surface corresponding to the set of spatialcoordinates to the set of modeling coordinates.

In certain embodiments, the updating step includes instructions that,when executed by the processor, cause the processor to: receive a set ofmodeling coordinates, wherein the set of modeling coordinates areexpressed in the robot coordinate system and define the surface of avolume of a patient anatomy; map (e.g., using surface matching,) thesurface corresponding to the set of spatial coordinates to the patientanatomy surface corresponding to the set of modeling coordinates (e.g.,by generating a transformation array or transformation matrix); andupdate the coordinate mapping based on the mapping of the surfacecorresponding to the set of spatial coordinates to the set of modelingcoordinates.

In one aspect, the disclosed technology includes a method of defining anoperational volume in which a surgical instrument attached to anend-effector of a robotic arm of a robotic surgical system can bemaneuvered, the method including the steps of: receiving, by a processorof a computing device, haptic feedback, from a force sensor attacheddirectly or indirectly to the robotic arm, prompted by movement of theend-effector (e.g., towards a patient); determining, by the processor,that the haptic feedback corresponds to contact of the instrument with amaterial (e.g., having a certain density or certain mechanicalproperties) (e.g., based at least on a magnitude of the haptic feedbackexceeding a threshold) (e.g., additionally based on the location of atleast one point on the instrument) (e.g., wherein the material is bone);determining, by the processor, a set of spatial coordinates, wherein theset of spatial coordinates includes a spatial coordinate for eachcontact of the instrument with the material, expressed using a robotcoordinate system (e.g., relative to the position of the end-effector),wherein each spatial coordinate corresponds to a point on the surface ofa volume (e.g., a point on a surface of a bone); receiving, by theprocessor, a model volume selected by a user (e.g., a model of a portionof bone to be removed), wherein the model volume is expressed in a robotcoordinate system; mapping, by the processor, the surface of the modelvolume to the set of spatial coordinates; generating, by the processor,an updated model volume, wherein coordinates of the updated model volumeare generated by converting coordinates of the model volume using themapping of the surface of the model volume to the set of spatialcoordinates; and storing, by the processor, the updated model volume.

In certain embodiments, the updated model volume is a constrainedoperational volume, wherein a terminal point of the surgical instrumentis temporarily constrained to within the constrained operational volume.

In certain embodiments, the model volume is generated from medical imagedata using a coordinate mapping.

In certain embodiments, the method includes receiving, by the processor,the updated model volume (e.g., a model of a portion of bone to beremoved), wherein the stored model volume is expressed in a first robotcoordinate system; receiving, by the processor, an updated coordinatemapping expressed in a second robot coordinate system; mapping, by theprocessor, the first robot coordinate system to the second robotcoordinate system; generating, by the processor, a second updated modelvolume by converting coordinates of the updated model volume to updatedcoordinates expressed in the second robot coordinate system using themapping between the first robot coordinate system and the second robotcoordinate system; and storing, by the processor, the second updatedmodel volume.

In one aspect, the disclosed technology includes a system for definingan operational volume in which a surgical instrument attached to anend-effector of a robotic arm of a robotic surgical system can bemaneuvered, the system including: a robotic arm with an end- effectorhaving an instrument attached thereto; a force sensor attached directlyor indirectly to the robotic arm (e.g., the force sensor located betweenthe instrument and the robotic arm); and a processor and a memory havinginstructions stored thereon, wherein the instructions, when executed bythe processor, cause the processor to: receive haptic feedback, from theforce sensor, prompted by movement of the end-effector (e.g., towards apatient); determine that the haptic feedback corresponds to contact ofthe instrument with a material (e.g., having a certain density orcertain mechanical properties) (e.g., based at least on a magnitude ofthe haptic feedback exceeding a threshold) (e.g., additionally based onthe location of at least one point on the instrument) (e.g., wherein thematerial is bone); determine a set of spatial coordinates, wherein theset of spatial coordinates includes a spatial coordinate for eachcontact of the instrument with the material, expressed using a robotcoordinate system (e.g., relative to the position of the end- effector),wherein each spatial coordinate corresponds to a point on the surface ofa volume (e.g., a point on a surface of a bone); receive a model volumeselected by a user (e.g., a model of a portion of bone to be removed),wherein the model volume is expressed in a robot coordinate system; mapthe surface of the model volume to the set of spatial coordinates;generate an updated model volume, wherein coordinates of the updatedmodel volume are generated by converting coordinates of the model volumeusing the mapping of the surface of the model volume to the set ofspatial coordinates; and store the updated model volume.

In certain embodiments, the updated model volume is a constrainedoperational volume, wherein a terminal point of the surgical instrumentis temporarily constrained to within the constrained operational volume.

In certain embodiments, the model volume is generated from medical imagedata using a coordinate mapping.

In certain embodiments, the instructions, when executed by theprocessor, cause the processor to: receive the updated model volume(e.g., a model of a portion of bone to be removed), wherein the storedmodel volume is expressed in a first robot coordinate system; receive anupdated coordinate mapping expressed in a second robot coordinatesystem; map the first robot coordinate system to the second robotcoordinate system; generate a second updated model volume by convertingcoordinates of the updated model volume to updated coordinates expressedin the second robot coordinate system using the mapping between thefirst robot coordinate system and the second robot coordinate system;and store the second updated model volume.

In one aspect, the disclosed technology includes a method of displayinga position of a surgical instrument attached to a robotic arm relativeto a patient anatomy for navigation during a robotically-assistedsurgical procedure, the method including: receiving, by a processor of acomputing device, a location of a terminal point of the surgical tool(e.g., wherein the location of the terminal point is determined, by theprocessor, using a known (e.g., stored) distance between a location ofthe robotic arm and the terminal point), wherein the location isexpressed in a robot coordinate system; receiving, by the processor, acoordinate mapping between the robot coordinate system and a medicalimage data coordinate system (e.g., a transformation array ortransformation matrix); converting, by the processor, using thecoordinate mapping, the location of the terminal point, such that theconverted location of the terminal point is expressed in a medical imagedata coordinate system; generating, by the processor, rendering dataincluding the converted location of the terminal point; and outputting,by the processor, the rendering data, wherein a display of the renderingdata includes a representation of the patient anatomy and arepresentation of the location of the terminal point, wherein asimulated distance between a point on the representation of the patientanatomy and the representation of the terminal point is proportional toa spatial distance between the terminal point and the correspondingpoint on the patient’s anatomy.

In certain embodiments, the rendering data corresponding to therepresentation of the patient anatomy is generated from medical imagedata coordinates generated from medical imaging data, wherein themedical image data coordinates are expressed in the medical image datacoordinate system.

In one aspect, the disclosed technology includes a system of displayinga position of a surgical instrument attached to a robotic arm relativeto a patient anatomy for navigation during a robotically-assistedsurgical procedure, the system including: a robotic arm with an end-effector having an instrument attached thereto; a force sensor attacheddirectly or indirectly to the robotic arm (e.g., the force sensorlocated between the instrument and the robotic arm); and a processor anda memory having instructions stored thereon, wherein the instructions,when executed by the processor, cause the processor to: receive alocation of a terminal point of the surgical tool (e.g., wherein thelocation of the terminal point is determined, by the processor, using aknown (e.g., stored) distance between a location of the robotic arm andthe terminal point), wherein the location is expressed in a robotcoordinate system; receive a coordinate mapping between the robotcoordinate system and a medical image data coordinate system (e.g., atransformation array or transformation matrix); convert, using thecoordinate mapping, the location of the terminal point, such that theconverted location of the terminal point is expressed in a medical imagedata coordinate system; generate rendering data including the convertedlocation of the terminal point; output the rendering data, wherein adisplay of the rendering data includes a representation of the patientanatomy and a representation of the location of the terminal point,wherein a simulated distance between a point on the representation ofthe patient anatomy and the representation of the terminal point isproportional to a spatial distance between the terminal point and thecorresponding point on the patient’s anatomy.

In certain embodiments, the rendering data corresponding to therepresentation of the patient anatomy is generated from medical imagedata coordinates generated from medical imaging data, wherein themedical image data coordinates are expressed in the medical image datacoordinate system.

In one aspect, the disclosed technology includes a method of performingvolume removal surgery with one or more instruments, wherein the one ormore instruments used by attaching to an end-effector of a robotic armof a robotic surgical system, the method including the steps of:registering a patient’s anatomy to express a model of the patient’sanatomy in a robot coordinate system; contacting, following removal of afirst volume of the patient’s anatomy, an instrument attached to theend-effector to the patient’s anatomy in a plurality of locations,wherein contact is determined by haptic feedback from a force sensorattached directly or indirectly to the robotic arm, wherein the hapticfeedback is prompted by movement of the end- effector (e.g., towards apatient); updating the model of the patient’s anatomy by determining aportion of the model that corresponds to the first volume of thepatient’s anatomy that has been removed using spatial coordinatescorresponding to the plurality of locations contacted; optionally,re-registering the patient’s anatomy by contacting a plurality ofre-registration locations with the instrument, wherein contact isdetermined by haptic feedback from a force sensor attached directly orindirectly to the robotic arm, wherein the haptic feedback is promptedby movement of the end-effector (e.g., towards a patient), andcoordinates in the model of the patient’s anatomy are converted suchthat they are expressed in a new robot coordinate system based on there-registration; defining an operational volume, wherein the operationalvolume is expressed in either the robot coordinate system or the newrobot coordinate system (e.g., if the re-registration step isperformed); and maneuvering the robotic arm such that a pre- definedterminal point on a surgical instrument is constrained to within theoperational volume for a period of time during a second volume removal.

In certain embodiments, the registering step includes: receiving, by aprocessor of a computing device, haptic feedback, from a force sensorattached directly or indirectly to the robotic arm, prompted by movementof the end-effector (e.g., towards a patient); determining, by theprocessor, that the haptic feedback corresponds to contact of theinstrument with a material (e.g., having a certain density or certainmechanical properties) (e.g., based at least on a magnitude of thehaptic feedback exceeding a threshold) (e.g., additionally based on thelocation of at least one point on the instrument) (e.g., wherein thematerial is bone); determining, by the processor, a set of spatialcoordinates, wherein the set of spatial coordinates includes a spatialcoordinate for each contact of the instrument with the material,expressed using the robot coordinate system, (e.g., relative to theposition of the end-effector), wherein each spatial coordinatecorresponds to a point on the surface of an anatomical volume (e.g., apoint on a surface of a bone); receiving, by the processor, a set ofmedical image data coordinates expressed using a medical image datacoordinate system that correspond to a patient anatomy surface (e.g.,determined from tomographic patient data (e.g., CT data, MRI data));mapping, by the processor, (e.g., using surface matching,) the surfacecorresponding to the set of spatial coordinates to the patient anatomysurface corresponding to the set of medical image data coordinates(e.g., by generating a transformation array or transformation matrix);generating, by the processor, a coordinate mapping between the robotcoordinate system and the medical image data coordinate system based onthe mapping between the surface corresponding to the set of spatialcoordinates and the surface corresponding to the set of medical imagedata coordinates; and storing, by the processor, the coordinate mapping(e.g., for navigational use by a surgeon during a surgical procedure).

In certain embodiments, the method including the step of: outputting, bythe processor, rendering data for display, wherein the rendering datacorresponds to a representation of a member’s position and at least aportion of the medical image data based on the coordinate mapping,wherein the member is selected from the group consisting of: theend-effector, the instrument, and a surgical instrument. In certainembodiments, the method including the steps of: generating, by theprocessor, new rendering data by modifying the rendering data based on achange in the end- effector’s position; and outputting, by theprocessor, the new rendering data for display.

In certain embodiments, a fiducial marker includes the material (e.g.,the end- effector contacts a fiducial marker with known size and shapesuch that the spatial coordinate is determined using a spatialrelationship between the fiducial marker and the patient’s anatomy).

In certain embodiments, the robotic arm is active and non-back drivable.

In certain embodiments, therobotic surgical system includes theprocessor.

In certain embodiments, the method includes storing, by the processor, apatient anatomy model wherein the patient anatomy model is defined bythe patient anatomy surface expressed in the robot coordinate system.

In certain embodiments, the updating step includes: determining, by theprocessor, a set of spatial coordinates, wherein the set of spatialcoordinates includes a spatial coordinate for each contact of theinstrument with the material, expressed using a robot coordinate system(e.g., relative to the position of the end-effector), wherein eachspatial coordinate corresponds to a point on the surface of ananatomical volume (e.g., a point on a surface of a bone); receiving, bythe processor, a set of medical image data coordinates that correspondto the surface of a volume of the patient’s anatomy, wherein eachmedical image data coordinate in the set of medical image datacoordinates is expressed using a medical image data coordinate system;receiving, by the processor, a coordinate mapping between the robotcoordinate system and the medical image data coordinate system (e.g., atransformation array or transformation matrix); determining, by theprocessor, one or more interior spatial coordinates in the set ofspatial coordinates that correspond to points inside the surface of thevolume of the patient’s anatomy based on the set of medical image datacoordinates and the coordinate mapping; determining, by the processor, aportion of the volume of the patient’s anatomy that has been removedusing the one or more interior spatial coordinates; generating, by theprocessor, a set of interior medical image data coordinates, wherein theset of interior medical image data coordinates includes an interiormedical image data coordinate for each of the one or more interiorspatial coordinates using the coordinate mapping; modifying, by theprocessor, the set of medical image data coordinates that define thesurface of the volume of the patient’s anatomy with the set of interiormedical image data coordinates such that first volume defined by the setof medical image data coordinates is larger than second volume definedby the modified set of medical image data coordinates; and storing, bythe processor, the modified set of medical image data coordinates (e.g.,for displaying to a surgeon).

In certain embodiments, the defining step includes: receiving, by aprocessor of a computing device, haptic feedback, from a force sensorattached directly or indirectly to the robotic arm, prompted by movementof the end-effector (e.g., towards a patient); determining, by theprocessor, that the haptic feedback corresponds to contact of theinstrument with a material (e.g., having a certain density or certainmechanical properties) (e.g., based at least on a magnitude of thehaptic feedback exceeding a threshold) (e.g., additionally based on thelocation of at least one point on the instrwnent) (e.g., wherein thematerial is bone); determining, by the processor, a set of spatialcoordinates, wherein the set of spatial coordinates includes a spatialcoordinate for each contact of the instrwnent with the material,expressed using a robot coordinate system (e.g., relative to theposition of the end-effector), wherein each spatial coordinatecorresponds to a point on the surface of a volume (e.g., a point on asurface of a bone); receiving, by the processor, a model volume selectedby a user (e.g., a model of a portion of bone to be removed), whereinthe model volume is expressed in a robot coordinate system; mapping, bythe processor, the surface of the model volume to the set of spatialcoordinates; generating, by the processor, an updated model volume,wherein coordinates of the updated model volume are generated byconverting coordinates of the model volume using the mapping of thesurface of the model volume to the set of spatial coordinates; andstoring, by the processor, the updated model volume.

In certain embodiments, the updated model volume is a constrainedoperational volume, wherein a terminal point of the surgical instrumentis temporarily constrained to within the constrained operational volume.

In certain embodiments, the model volume is generated from medical imagedata using a coordinate mapping.

In certain embodiments, the method includes receiving, by the processor,the updated model volume (e.g., a model of a portion of bone to beremoved), wherein the stored model volume is expressed in a first robotcoordinate system; receiving, by the processor, an updated coordinatemapping expressed in a second robot coordinate system; mapping, by theprocessor, the first robot coordinate system to the second robotcoordinate system; generating, by the processor, a second updated modelvolume by converting coordinates of the updated model volume to updatedcoordinates expressed in the second robot coordinate system using themapping between the first robot coordinate system and the second robotcoordinate system; and storing, by the processor, the second updatedmodel volume.

In certain embodiments, the re-registering step includes: receiving, bya processor of a computing device, haptic feedback, from a force sensorattached directly or indirectly to the robotic arm, prompted by movementof the end-effector (e.g., towards a patient); determining, by theprocessor, that the haptic feedback corresponds to contact of theinstrument with a material (e.g., having a certain density or certainmechanical properties) (e.g., based at least on a magnitude of thehaptic feedback exceeding a threshold) (e.g., additionally based on thelocation of at least one point on the instrument) (e.g., wherein thematerial is bone); determining, by the processor, a set of spatialcoordinates, wherein the set of spatial coordinates includes a spatialcoordinate for each contact of the instrument with the material,expressed using the robot coordinate system (e.g., relative to theposition of the end-effector), wherein each spatial coordinatecorresponds to a point on the surface of an anatomical volume (e.g., apoint on a surface of a bone); receiving, by the processor, a coordinatemapping between a robot coordinate system and a medical image datacoordinate system (e.g., a transformation array or transformationmatrix), wherein the robot coordinate system corresponds to a physicalcoordinate system of the end- effector; updating, by the processor, thecoordinate mapping based on a mapping of the surface corresponding tothe set of spatial coordinates; and storing, by the processor, theupdated coordinate mapping (e.g., to provide an accurate navigationalmodel for use by a surgeon during a surgical procedure).

In certain embodiments, the mapping is generated using surface matching.

In certain embodiments, the updating step includes: determining, by theprocessor, a set of modeling coordinates, by converting, using thecoordinate mapping, a set of medical image modeling coordinates definingthe surface of a volume of a patient anatomy, wherein the set ofmodeling coordinates are expressed in the robot coordinate system anddefine an anticipated location of the surface of the volume, and the setof medical image modeling coordinates have been generated from medicalimaging data; and mapping, by the processor, (e.g., using surfacematching,) the surface corresponding to the set of spatial coordinatesto the patient anatomy surface corresponding to the set of modelingcoordinates (e.g., by generating a transformation array ortransformation matrix); and updating, by the processor, the coordinatemapping based on the mapping of the surface corresponding to the set ofspatial coordinates to the set of modeling coordinates.

In certain embodiments, the updating step includes: receiving, by theprocessor, a set of modeling coordinates, wherein the set of modelingcoordinates are expressed in the robot coordinate system and define thesurface of a volume of a patient anatomy; mapping, by the processor,(e.g., using surface matching,) the surface corresponding to the set ofspatial coordinates to the patient anatomy surface corresponding to theset of modeling coordinates (e.g., by generating a transformation arrayor transformation matrix); and updating, by the processor, thecoordinate mapping based on the mapping of the surface corresponding tothe set of spatial coordinates to the set of modeling coordinates.

In another aspect, the disclosed technology includes a method ofupdating an operational volume in which a surgical instrument attachedto an end-effector of a robotic arm of a robotic surgical system can bemaneuvered, the method including the steps of: receiving, by theprocessor, a stored model volume including coordinates (e.g., a model ofa portion of bone to be removed), wherein the stored model volume isexpressed in a first robot coordinate system; receiving, by theprocessor, an updated coordinate mapping expressed in a second robotcoordinate system; converting, by the processor, each coordinate of thestored model volume to be expressed in the second robot coordinatesystem using the updated coordinate mapping; and storing, by theprocessor, an updated model volume including the converted coordinates.

In order for the present disclosure to be more readily understood,certain terms used herein are defined below. Additional definitions forthe following terms and other terms may be set forth throughout thespecification.

In this application, the use of “or” means “and/or” unless statedotherwise. As used in this application, the term “comprise” andvariations of the term, such as “comprising” and “comprises,” are notintended to exclude other additives, components, integers or steps. Asused in this application, the terms “about” and “approximately” are usedas equivalents. Any numerals used in this application with or withoutabout/approximately are meant to cover any normal fluctuationsappreciated by one of ordinary skill in the relevant art. In certainembodiments, the term “approximately” or “about” refers to a range ofvalues that fall within 25%, 20%, 19%, 18%, 17%, 16%, 15%, 14%, 13%,12%, 11%, 10%,9%,8%, 7%,6%,5%,4%, 3%, 2%, 1%, or less in eitherdirection (greater than or less than) of the stated reference valueunless otherwise stated or otherwise evident from the context (exceptwhere such number would exceed 100% of a possible value).

Mapping: As used herein, “mapping” refers to establishing a functionbetween two sets of coordinates or data corresponding to two sets ofcoordinates. The function between the two sets may be discrete orcontinuous. A mapping allows coordinates recorded and/or stored in onecoordinate system to be converted to coordinates in another coordinatesystem and vice versa. Two sets of coordinates expressed in the samecoordinate system may be mapped with each other as well. A map ormapping may be stored on a computer readable medium as an array ormatrix of data. In certain embodiments, a map or mapping is a lineartransform stored as an array on a computer readable medium. In certainembodiments, the map or mapping is used to convert between coordinatesystems. In certain embodiments, the coordinate systems are Cartesian.In some embodiments, at least one of the coordinate systems isnon-Cartesian. A mapping may be an optimized function, wherein themapping represents the function of minimal error or error below athreshold according tothe mapping method (e.g., surface matching). Incertain embodiments, mapping comprises surface matching. Herein, “a map”and “a mapping” are used interchangeably.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Drawings are presented herein for illustration purposes, not forlimitation. The foregoing and other objects, aspects, features, andadvantages of the invention will become more apparent and may be betterunderstood by referring to the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 is an illustration of a robotic surgical system in an operatingroom, according to an illustrative embodiment of the invention;

FIG. 2 is an illustration of the use of a robotic surgical system withrobot-based navigation in a surgical procedure, according to anillustrative embodiment of the invention;

FIG. 3 is an illustration of a process of using a robotic surgicalsystem to assist with a surgical procedure, according to an illustrativeembodiment of the invention;

FIG. 4 is an illustration of various options for an imaging process,according to an illustrative embodiment of the invention;

FIGS. 5A, 5B, and 5C are illustrations of force sensor implementations,according to illustrative embodiments of the invention;

FIG. 6 is an illustration of a surgical instrument, according to anillustrative embodiment of the invention;

FIGS. 7A through 7D are illustrations of implementations of a forcesensor integrated in a surgical drill, according to an illustrativeembodiments of the invention;

FIG. 8 is an illustration of a process for determining a position of atarget anatomy, according to an illustrative embodiment of theinvention;

FIG. 9 is an illustration of an example mechanical marker, according toan illustrative embodiment of the invention;

FIG. 10 is an illustration of a process for determining a position of atarget anatomy based on automatic reregistration, according to anillustrative embodiment of the invention;

FIG. 11 is an illustration of a process for rendering feedback to anoperator, according to an illustrative embodiment of the invention;

FIG. 12 is an illustration of a process for tracking an instrument,according to an illustrative embodiment of the invention;

FIG. 13 is an illustration of a process for guiding an instrument,according to an illustrative embodiment of the invention;

FIGS. 14 to 35 are illustrations of a patient’s anatomy and a surgicalrobotic system during a method of performing a surgical procedure on thepatient’s spine using the robotic surgical system and robot navigation,according to an illustrative embodiment of the invention.

FIG. 36 illustrates a block diagram of an exemplary cloud computingenvironment, according to an illustrative embodiment of the invention;

FIG. 37 is a block diagram of a computing device and a mobile computingdevice, according to an illustrative embodiment of the invention; and

FIG. 38 is an illustration of an example robotic surgical system,according to an illustrative embodiment of the invention.

DETAILED DESCRIPTION

It is contemplated that systems, devices, methods, and processes of theclaimed invention encompass variations and adaptations developed usinginformation from the embodiments described herein. Adaptation and/ormodification of the systems, devices, methods, and processes describedherein may be performed by those of ordinary skill in the relevant art.

Throughout the description, where articles, devices, and systems are described as having, including, or comprising specific components, orwhere processes and methods are described as having, including, orcomprising specific steps, it is contemplated that, additionally, thereare articles, devices, and systems of the present invention that consistessentially of, or consist of, the recited components, and that thereare processes and methods according to the present invention thatconsist essentially of, or consist of, the recited processing steps.

It should be understood that the order of steps or order for performingcertain action is immaterial so long as the invention remains operable.Moreover, two or more steps or actions may be conducted simultaneously.

The mention herein of any publication, for example, in the Backgroundsection, is not an admission that the publication serves as prior artwith respect to any of the claims presented herein. The Backgroundsection is presented for purposes of clarity and is not meant as adescription of prior art with respect to any claim. Headers are providedfor the convenience of the reader and are not intended to be limitingwith respect to the claimed subject matter.

FIG. 1 illustrates an example robotic surgical system in an operatingroom 100. In some implementations, one or more surgeons, surgicalassistants, surgical technologists and/or other technicians, (106 a-c)perform an operation on a patient 104 using a robotic-assisted surgicalsystem. In the operating room the surgeon may be guided by the roboticsystem to accurately execute an operation. This may be achieved byrobotic guidance of the surgical tools, including ensuring the propertrajectory of the tool (e.g., drill or screw). In some implementations,the surgeon defines the trajectory intra-operatively with little or nopre- operative planning. The system allows a surgeon to physicallymanipulate the tool holder to safely achieve proper alignment of thetool for performing crucial steps of the surgical procedure. Operationof the robot arm by the surgeon (or other operator) in force controlmode permits movement of the tool in a measured, even manner thatdisregards accidental, minor movements of the surgeon. The surgeon movesthe tool holder to achieve proper trajectory of the tool (e.g., a drillor screw) prior to operation or insertion of the tool into the patient.Once the robotic an is in the desired position, the arm is fixed tomaintain the desired trajectory. The tool holder serves as a stable,secure guide through which a tool may be moved through or slid at anaccurate angle. Thus, the disclosed technology provides the surgeon withreliable instruments and techniques to successfully perform his/hersurgery.

In some embodiments, the operation may be spinal surgery, such as adiscectomy, a foraminotomy, a laminectomy, or a spinal fusion. In someimplementations, the surgical robotic system includes a surgical robot102 on a mobile cart. The surgical robot 102 may be positioned inproximity to an operating table 112 without being attached to theoperating table, thereby providing maximum operating area and mobilityto surgeons around the operating table and reducing clutter on theoperating table. In alternative embodiments, the surgical robot (orcart) is securable to the operating table. In certain embodiments, boththe operating table and the cart are secured to a common base to preventany movement of the cart or table in relation to each other, even in theevent of an earth tremor.

The mobile cart may permit a user (operator) 106 a, such as atechnician, nurse, surgeon, or any other medical personnel in theoperating room, to move the surgical robot 102 to different locationsbefore, during, and/or after a surgical procedure. The mobile cartenables the surgical robot 102 to be easily transported into and out ofthe operating room 100. For example, a user 106 a may move the surgicalrobot into the operating room from a storage location. In someimplementations, the mobile cart may include wheels, a track system,such as a continuous track propulsion system, or other similar mobilitysystems for translocation of the cart. The mobile cart may include anattached or embedded handle for locomotion of the mobile cart by anoperator.

For safety reasons, the mobile cart may be provided with a stabilizationsystem that may be used during a surgical procedure performed with asurgical robot. The stabilization system increases the global stiffnessof the mobile cart relative to the floor in order to ensure the accuracyof the surgical procedure. In some implementations, the wheels include alocking system that prevents the cart from moving. The stabilizing,braking, and/or locking system may be activated when the machine isturned on. In some implementations, the mobile cart includes multiplestabilizing, braking, and/or locking systems. In some implementations,the stabilizing system is electromechanical with electronic activation.The stabilizing, braking, and/or locking system(s) may be entirelymechanical. The stabilizing, braking, and/or locking system(s) may beelectronically activated and deactivated.

In some implementations, the surgical robot 102 includes a robotic armmounted on a mobile cart. An actuator may move the robotic arm. Therobotic arm may include a force control end-effector configured to holda surgical tool. The robot may be configured to control and/or allowpositioning and/or movement of the end-effector with at least fourdegrees of freedom (e.g., six degrees of freedom, three translations andthree rotations).

In some implementations, the robotic arm is configured to releasablyhold a surgical tool, allowing the surgical tool to be removed andreplaced with a second surgical tool. The system may allow the surgicaltools to be swapped without re-registration, or with automatic or semi-automatic re-registration of the position of the end-effector.

In some implementations, the surgical system includes a surgical robot102, a tracking detector 108 that captures the position of the patientand different components of the surgical robot 1 02, and a displayscreen 110 that displays, for example, real time patient data and/orreal time surgical robot trajectories.

In some implementations, a tracking detector 108 monitors the locationof patient 104 and the surgical robot 102. The tracking detector may bea camera, a video camera, an infrared detector, field generator andsensors for electro-magnetic tracking or any other motion detectingapparatus. In some implementation, based on the patient and robotposition, the display screen displays a projected trajectory and/or aproposed trajectory for the robotic arm of robot 102 from its currentlocation to a patient operation site. By continuously monitoring thepatient and robotic arm positions, using tracking detector 108, thesurgical system can calculate updated trajectories and visually displaythese trajectories on display screen 110 to inform and guide surgeonsand/or technicians in the operating room using the surgical robot. Inaddition, in certain embodiments, the surgical robot 102 may also changeits position and automatically position itself based on trajectoriescalculated from the real time patient and robotic arm positions capturedusing the tracking detector 108. For instance, the trajectory of theend- effector can be automatically adjusted in real time to account formovement of the vertebrae or other part of the patient during thesurgical procedure. The disclosed technology includes a robot-basednavigation system for real-time, dynamic re-registration of a patientposition (e.g., position of vertebrae of a patient) during a procedure(e.g., surgical procedure, e.g., a spinal surgery). An example roboticsurgical system is shown in FIG. 38 . The robotic surgical system 3800includes a robotic arm 3802. The robotic arm can have 3, 4, 5, 6, or 7degrees of freedom. The robotic arm 3802 has an end effector 3804.

In certain embodiments, the robotic arm 3802 includes a position sensor3806 for dynamically tracking a position of the end effector 3804 and/orsurgical instrument during a surgical procedure. Additionally, one ormore points of the surgical instrument can be dynamically tracked, forexample, at a rate of at least 100 Hz, 250 Hz or greater, 500 Hz orgreater, or 1000 Hz or greater (e.g., position determination persecond).

In certain embodiments, the system 3800 includes a force feedbacksubsystem 3808. The force feedback subsystem 3808 can include sensor(s),actuator(s), controller(s), servo(s), and/or other mechanisms fordelivering a haptic force to a user manipulating the end effector or asurgical instrument inserted in the instrument holder of the endeffector. The force feedback subsystem 3808 can detect the resistiveforce caused by the surgical instrument contacting, moving against,penetrating, and/or moving within a tissue of the patient. Furthermore,the force feedback subsystem 3808 can distinguish between contactedtissue types (e.g., determining when contacted tissue meets or exceeds athreshold resistance, e.g., when the tissue is bone).

The force feedback subsystem 3808 can also detect a force delivered bythe operator. For example, it can detect forces delivered by directmanipulation of the surgical instrument inserted in the surgicalinstrument holder of the end effector to cause movement of the surgicalinstrument and, therefore, the end effector. The force feedbacksubsystem 3808 can further distinguish between the force delivered bythe operator and the resistive force caused by movement of the surgicalinstrument in relation to the tissue of the patient. This allows theoperator to both apply forces to the system as well as feel resistance(e.g., via haptic feedback) as a surgical instrument contacts tissue inthe patient. [0108] In certain embodiments, the robotic surgical system3800 includes a display 3810 that is attached to, embedded within, orotherwise positioned in relation to the robotic arm being directlymanipulated by the operator (e.g., surgeon) to allow for unimpededvisual feedback to the operator during theprocedure.

When an operator uses the system, the system initially accesses (e.g.,and graphically renders on the display) an initial registration of atarget volume, such as a vertebra of the patient. This can beaccomplished using medical images of the patient, including MRI, CT,X-rays, SPECT, ultrasound, or the like. These images can be obtainedpreoperatively or intraoperatively.

As the operative moves the position of the end effector, the position ofthe end effector is dynamically determined (e.g.,byprocessor3812).Specifically, in some implementations, the system dynamically determinesa 3D position of one or more points of a surgical instrument.

Forces received by the surgical instrument are dynamically determinedwhen the surgical instrument contacts, moves against, penetrates, and/ormoves within the patient. The system can measure these forces anddistinguish between contacted tissue types. This can be accomplished,for example, by determining when contacted tissue meets or exceeds athreshold resistance, such as when the tissue is bone). The system canfurther detect forces applied to the surgical instrument by the operatorand distinguish between forces delivered by the operator and theresistive force caused by movement of the surgical instrument inrelation to the tissue of the patient.

In certain embodiments, the system can dynamically re-register thepatient position based at least in part on an updated position of theend effector determined by the position sensor. This can be used toupdate the 3D representation of the patient situation based at least inpart on the updated position of the end effector when it is determined(e.g., via the force feedback subsystem) that the surgical instrument isin contact with a target anatomy. This can be accomplished using asurface matching algorithm keyed to the initial (or previous)registration.

For example, the system can dynamically re-register the patient positionupon detected contact or proximity of the end effector, or the surgicalinstrument, or a portion or component of the surgical instrument or endeffector, with a pre-planned fiducial, such as a mechanical marker, amarker fixed to the patient. Alternatively, the system can dynamicallyre-register the patient position based upon the updated position of theend effector determined upon operator command, such as the operatorpressing a button or otherwise activating a graphical or tactile userinterface when a re-registered representation is desired.

A surgical instrument holder can be connected to the end effector forinsertion or attachment of a surgical instrument therein/thereto. Theinstrument holder can be removable. In such instances, attachment of theinstrument holder to the end effector is precise and predictable suchthat it is always connected in the same position.

The robotic arm is designed to allow direct manipulation of a surgicalinstrument by an operator (e.g., by a surgeon) when the surgicalinstrument is inserted in/attached to the surgical instrument holder ofthe end effector. The manipulation of the instrument can be subject tohaptic constraints based on the position of the end effector (and/or thesurgical instrument) in relation to the patient. The surgical instrumenthas a known geometry and position in relation to the surgical instrumentholder such that the location of the instrument (e.g., the tip of theinstrument) is known by the robotic surgical system. For example, when asurgical instrument is fully inserted into the instrument holder, theposition of the instrument is known to the robotic surgical systembecause the position of the end effector is known as well as informationabout the surgical instrument and the instrument holder.

In certain embodiments, a tool center point (TCP) facilitates precisepositioning and trajectory planning for surgical instrument guides andsurgical instruments inserted through or attached to the surgicalinstrument holder. Surgical instruments can be engineered such that wheninserted into the surgical instrument holder, there is a defined toolcenter point with known coordinates relative to robotic arm. The originof a coordinate system used to define the tool center point may belocated at a flange of a robotic arm. It may additionally be located atany convenient to define point such as an interface, joint, or terminalaspect of a component of a robotic surgical system.

In certain embodiments, because the TCP is in a constant positionrelative to the robotic arm, regardless of whether a surgical guide orsurgical instrument is being used with the surgical instrument holder, asurgeon can be provided visualization of the orientation, trajectory,and position of an instrument or instrument guide used with the surgicalinstrument holder. The use of engineered surgical instrument systemseliminates the need for navigation markers to be attached to the end ofsurgical guides or tools in order to precisely determine the position,orientation, and trajectory of a surgical instrument guide relative to apatient’s anatomy.

Additionally, a navigation marker attached to surgical instrument holdercan be used to track the position and orientation of the universalsurgical instrument guide to update the position, orientation, andcurrent trajectory based on manipulation of robotic arm by a surgeon.Additional information provided by patient imaging (e.g., CT data, radioimaging data, or similar) taken pre- or intra-operatively as well asnavigation markers attached to a patient’s body may be combined withdata from a navigation marker attached to a universal surgicalinstrument guide and displayed on a screen viewable by the surgeon suchthat the surgeon can see the location of necessary features of thepatient’s anatomy and the position, trajectory, and orientation of asurgical instrument or surgical instrument guide relative to saidanatomy.

FIGS. 5A-C illustrate example locations for mounting a force sensor(e.g., force/torque sensor 430). In some implementations, as shown inFIG. 5A, the force sensor 502 a is located between the tool holder 506an and robot 504 a. Using this configuration, the sterile cover 508 amay be wrapped around the robot arm and between the force sensor and thetool holder to ensure sterilization. The force sensor 502 a may providefor direct measurement of forces (e.g., forces and/or torques) on thetool. The force sensor 502 a may be designed to resist flexing. Theforce sensor 502 a may be designed to flex under the stress of certainexternal forces. The displacement caused when an external force isapplied may be calculated based on the force and/or torque applied tothe tool, radial force stiffness, axial torque stiffness, and thediameter of the holder to which the tool is attached.

As shown in FIGS. 5B and 5C, respectively, the force sensor (e.g., 502bin FIG. 5B or 502 c in FIG. 5C) may be located on the robot or the toolholder, respectively. These configurations may exclusively measure theforces and/or torques applied by the user. The force sensor 508 may beconnected to the robot with an intermediary analog box which measuresforces and torques and transmits them via a network (e.g., Ethernet,CAN, wireless, internet, private LAN, public LAN, etc.). Combinations ofthe above mentioned force sensor positions are possible to achievepre-defined behavior (e.g. the first sensor in the base FIG. 5A and thesecond one in the handle FIG. 5B may be positioned to allow the feedbackcontrol system to decouple forces applied to the surgical tool fromforces and/ortorque applied by a user).

Additionally, in some implementations the force sensor is integrateddirectly in the surgical instrument. For example, the force sensor maybe integrated directly in the surgical drill bit as illustrated in FIG.6 . While the implementation of the force sensor 604 is described inrelation to a drill bit 602 as shown in FIG. 6 , the force sensor 604may be similarly integrated in other surgical instruments. Integratingthe force sensor 604 in a surgical instrument, such as a drill bit 602,may be more robust as it minimizes the impact of external disturbancesfor measuring forces applied to the drill bit.

In the example configuration shown in FIG. 6 , the force sensor 604 isintegrated in the shaft of the drill bit 602. The force sensor 604, insome implementations, is located on the drill bit 602 outside of thebody 610 of the drill as shown in FIG. 6 . In other implementations, theforce sensor 604 is located inside the body 610 of the drill, therebybetter protecting the force sensor 604 from external influences. Forcesensor can have multiple degrees of freedom and measure, for example, 1to 3 forces and/or 1 to 3 torques. Forces are transmitted from therotating shaft through a connector 606. The connector, in someimplementations, is one or more brushes that provide an electricalconnection to the force sensor 604. If the force sensor is an opticalsensor, the connector may be an optical transmitter (e.g. LED) and/oroptical receiver (e.g., photodiode). In this example, the brushescontact the drill bit thereby forming an electrical connection with theforce sensor 604. In some implementations, the brushes touch one or morecontacts on the drill bit to form the electrical connection.

An electric or pneumatic motor 608 rotates the drill bit 602 shaft. Insome implementations, a sensor 612 (e.g., an encoder) measures positionof the shaft. The sensor 612 measures the position of the shaft in orderto correlate forces measured by the force sensor to the relativeposition of the shaft. For example, if the force sensor is located in adrill bit, the measurement of the direction of the force will vary asthe drill bit rotates. Specifically, the force sensor measures force andthe direction of the force periodically (e.g., every millisecond, everymicrosecond, or somewhere therebetween). The drill bit rotates as thesurgeon pushes it into bone. When the drill contacts the bone, the forcesensor will indicate some force (F1) in a direction (D1). One periodlater (e.g., one millisecond), the drill bit will rotate slightly so theforce sensor will indicate force of the same value (F1) (assuming aconstant force is applied) in a different direction (D2). The directionof the force will continue to change relative to a single perspective asthe drill bit rotates even if surgeon pushes into the bone with aconstant force. A constantly changing force direction is not acceptable.In order to correlate the directions (e.g., D1, D2) with the globaldirection of the force (D) coming from the bone (seen by the surgeon,robotic system etc.) the position of the drill in the global space mustbe calculated as the drill bit rotates. The sensor 612 is used tomeasure the position of the shaft and thus determine the globaldirection of the force (D). The sensor 612 may be located on the back ofthe motor 608 as shown in FIG. 6 . The sensor 612 may be located inother locations relative to the motor 608 as well.

The force sensor 604 may be provided in various configurations as shownin FIGS. 7A-D. In each configuration, the goal is to measure forces onthe tip of the tool (e.g., drill bit ultrasound bit, etc.). In theexample shown in FIG. 7A the force sensor 604 is integrated in the shaftof the drill bit 602 as described in relation to FIG. 6 . The forcesensor 604 may communicate with a connector 606 (shown in FIG. 6 ) via asensor cable 702. The sensor cable 702, in some implementations, isrouted inside the drill bit 602. In some implementations, the connector606 (shown in FIG. 6 ) is electrically connected to the sensor cable 702via one or more connection pads.

The force sensor 604 in this example may be a miniaturized industrialsensor (e.g., the multi-axis force/torque sensor from ATI IndustrialAutomation, Inc. of Apex, N.C.) that measures, for example, all sixcomponents of force and torque using a transducer. Alternatively, theforce sensor 604 may be an optical sensor. Alternatively, the forcesensor 604 may comprise a strain gauge 706 integrated directly into theshaft of the drill bit 602 as shown in FIG. 7B.

As shown in FIG. 7C, the force sensor 604, in some implementations,measures forces on the motor instead of measuring forces on the drillbit 602 itself. As shown in FIG. 7D, the shaft of the drill bit 602, insome implementations, includes a flexible element 708 that allows thedrill bit 602 to bend (e.g., only slightly) such that after deflectionof the shaft of the drill bit 602, forces can be measured by the forcesensor 604. In some implementations, for the configuration shown inFIGS. 7C and 7D, the measurement of shaft positions (e.g., by sensor 612as shown in FIG. 6 ) may be omitted as the forces are measured directlyin the instrument coordinate frame.

A goal of the robot-based navigation is to assist a surgeon during asurgical procedure that results in a change to patient’s target anatomy.The implants and surgical instruments are used for this purpose and therobotic system assists the surgeon to improve the accuracy with whichthese instruments are used during the surgical procedure. FIG. 2 shows aschematic of all the system that assist in surgery, in some embodiments.

The interaction of a surgeon and components of a surgical system isfurther outlined in FIG. 3 . In reference to FIG. 3 , before thesurgical procedure begins, the patient’s target anatomy medical imagesare obtained using an appropriate imaging technique (e.g., which can beused to generate a model of the patient’s anatomy) and used in thefollowing processes.

The process of determining target anatomy position takes as an inputtarget anatomy. The target anatomy may be modeled using medical images,wherein medical images are taken using a medical imaging technique orscanning-based technique. As a result, determining the target anatomyposition provides the exact anatomy position of the target anatomy atany moment in time. The process of rendering feedback providesinformation to the surgeon based on medical images, anatomy position andinstrument(s) position. The rendering is visually displayed to thesurgeon. The tracking instruments process takes surgical instruments andas a result calculates their position. At the same time the instrumentis guided which means that their spatial position is constrained in someway by the robotic system (i.e., using an operational volume). Therobotic system implements all these four processes and the surgeonparticipates in the two of them: guidance andrendering feedback.

Different options for the imaging process are shown in FIG. 4 . The goalof the imaging process is to obtain representation of the patient’starget anatomy. In some implementations, imaging is used to obtain therepresentation of the patient’s target anatomy. The images can beobtained pre- operatively using various modalities, such as MRI, CT, orX- Rays. Alternatively, the images can be obtained intra-operativelyusing intra-operative imaging techniques, such as using flat-panelfluoroscopy technology (e.g. the 0-Arm by Medtronic of Minneapolis,Minnesota), intra-operative CTs, MRI and ultrasound. For example,intra-operative images can be captured using an intra- operativefluoroscopy device (e.g., a C-Arm).

There are other ways of obtaining information about the target anatomy.For example, information about the target anatomy can be collected bysurface scanning using a haptic device and force feedback. Using such adevice mounted on the robot allows user to measure forces that canprovide spatial information about surface and rigidity of tissues.

Alternatively known techniques of surface scanning, such as a laserscanner, can be used. Additionally, visual, direct exploration can beused to explore the patient’s anatomy. The outcome of these techniques,if used, is stored as medical image data, which is used to model thepatient’s anatomy.

FIG. 8 illustrates a range of processes that can be used for definingthe position of the target anatomy (i.e., registration). Typically, theposition of the target anatomy is managed using a registration procedureto identify the position of the anatomy in reference to an anatomy-fixed marker and later by the tracking (e.g., optical, orelectro-magnetic) the marker and assuming that the marker movesappropriately with the anatomy to determine the position of the targetanatomy. As discussed above in the Background, there are disadvantagesto this approach. In certain embodiments, the disclosed technologyutilizes force-based anatomy registration and re-registration to addressthese shortcomings. In-between re-registrations robot encoders can beused to immediately obtain instrument position at a high frequency(e.g., from 200 to 500 Hz, 500 to 800 HZ, or 800 Hz to 1200 Hz).

The initial registration data can be used along with a stability moduleto manage the location of the end-effector relative to the targetanatomy. The stability module can run all the time in the background(e.g., automatically), in certain embodiments, without surgeon takingany special actions.

Examples of such a stability module, includes surface matching methodswhich take a set of points (e.g. measured points) and finds the bestmatch between the set of points and another set of points (e.g., fromthe surface of the vertebra on medical images). Example algorithm forsurface matching is Iterative Closet Point method described in Section4.5.3 of “A Robotic System for Cervical Spine Surgery”, SzymonKostrzewski, Warsaw University of Technology (2011).

In certain embodiments, other algorithms are used for a stabilitymodule. Re-registration can be accomplished using other methods as well.For example, in certain embodiments, re-registration can be accomplishedby periodically re-validating using fiducials. In certain embodiments,when necessary, the surgeon can touch pre-placed fiducials andre-register with an instrument attached to a robotic arm.

In certain embodiments, a specially designed marker can be fixed to thepatient as shown in FIG. 9 . This marker has a set of conic holes whichcan be easily found by a robot having a force sensor. For example, asurgeon can push a button for the robot to re-register. Then the robotautomatically or the surgeon manually would bring the robot to the holesin the marker. After identifying at least 3 holes the re-registrationcan be found and the surgeon can continue with the surgery.

FIG. 10 is a schematic view of a process of determining a position basedon automatic re-registration. The re-registration method of FIG. 10utilizes initial registration data, patient medical images andinformation about the target anatomy coming from the robotic system(e.g., measured points on the surface of the bone) and, based on thisinformation, updates the registration data which is used by the anatomytracking process to provide the position of the target anatomy.

Various ways of rendering feedback are shown in FIG. 11 . For example,feedback can be rendered using screens accessible in the operating room.In one example, feedback can be rendered using a robot-held userinterface, such as the user interface described in U.S. Pat. ApplicationNo. 14/858,325, filed Sep. 18, 2015, entitled “Robot-Mounted UserInterface for Interacting with Operation Room Equipment”, the content ofwhich is hereby incorporated by reference in its entirety.

FIG. 12 is an illustration of exemplary processes for trackinginstruments. In some embodiments, tracking can be accomplished usingmarkers attached to the instrument and optical/electro-magnetictrackers. In certain embodiments, the instruments are tracked using therobotic system which provides their position in space. The robot “knows”the instrument position because it has a measurement system that is usedto determine the position of the position of the end effector. Coupledwith a known geometry of a given instrument, the position of theinstrument attached to the end effector (e.g., in a predicable manner)is known by the robotic surgical system. Mechanical template trackingrefers to mechanical, custom-made templates, which fit in one positiononly on the top of the target anatomy and contain guide for guidinginstruments. Passive arms are standard surgical arms, which can be fixedin pre-defined position in the operating room.

FIG. 13 shows exemplary systems for guiding instruments during surgicalprocedures. In certain embodiments, surgical instruments are guided withrobotic guidance. In certain embodiments, robotic guidance occursautomatically (i.e., without input from a surgeon) after registration ofthe patient’s anatomy with the robotic surgical system. In certainembodiments, manual guidance and passive guidance are used during somestages of surgical procedures. Uncertain embodiments, manual guidanceand passive guidance are used in all stages of surgical procedure.

A robotic surgical system with instrument attached can be used tocontact a patient’s anatomy at a plurality of contact points determinedusing haptic feedback from a force sensor attached directly orindirectly to the robotic an of the robotic surgical system. Thecoordinates of the plurality of contacts define a set of spatialcoordinates. The value of a spatial coordinate is determined by storingthe position of a portion of the robotic surgical system (e.g., aterminal point of an instrument or surgical instrument or the roboticarm) in the robot’s coordinate system when contact is determined.

A set of spatial coordinates recorded from contact of the instrumentwith the patient’s anatomy can be used to perform many navigational andsurgical guidance functions such as registration, modeling volumeremoval, re-registration, defining operational volumes, revisingoperational volumes after re-registration, converting stored volumemodels to physical locations, and displaying surgical instrumentsrelative to a patient’s anatomy on navigation screens. A processor thatis either a part of the robotic surgical system or part of a remotecomputing device (e.g., on a server) can be used to correlate thecoordinates of surgical instruments, instruments, and/or a patient’sanatomy by generating and using appropriate coordinate mappings incombination with sets of spatial coordinates. In some embodiments, a setof spatial coordinates may be provided for further use, wherein the setof spatial coordinates are generated using a technique other thanhaptic-feedback- based contacting of the patient’s anatomy with aninstrument attached to a robotic arm. For example, the set of spatialcoordinates may be provided as a result of a known registrationtechnique.

A patient’s anatomy can be registered with a robotic surgical systemwithout the use of a separate navigation system by contacting aninstrument to the patient’s anatomy to generate a set of spatialcoordinates that can be correlated with a model of the patient generatedusing medical imaging data. In certain embodiments, a surgeon uses asurgical instrument to contact the patient, including duringregistration. In some embodiments, one or more navigation markers areused for reference during registration. Once the processor hasdetermined a set of spatial coordinates, wherein each spatial coordinatecorresponds to a point on the surface of an anatomical volume (e.g., apatient’s vertebrae), the set of spatial coordinates can be mapped witha model of the patient’s anatomy. The patient’s anatomy may be modeledusing medical imaging data. In certain embodiments, CT, MRI, or x-raydata is used to pre-operatively generate a 3D model of the patient’sanatomy. In some embodiments, medical images used to generate patientanatomy models are taken intra-operatively.

A set of spatial coordinates is mapped to a model of a patient’s anatomyby determining a function for converting points in the model tocoordinates in the robot coordinate system (i.e., physical reality) andvice versa. In certain embodiments, mappings are made using surfacematching of a surface defined by a set of spatial coordinates and thesurface of the anatomical model. Points on the surface of the patient’sanatomy or the anatomical model of the patient’s anatomy may beextrapolated from known points (i.e., points measured by contacting thepatient’s anatomy with an instrwnent or data points collected duringmedical imaging). The extrapolated points may be used to generate themapping. An example of a surface matching method is Iterative ClosestPoint (ICP) described in Section 4.5.3 of “A Robotic System for CervicalSpine Surgery,” Szymon Kostrzewski, Warsaw University of Technology(2011). The contents of Section 4.5.3 are hereby incorporated byreference herein in their entirety. In general, any algorithm or methodthat generates a function that can be used to transform points from onecoordinate system to another (i.e., bilinear transform) is appropriatefor use. A threshold may be specified that defines an error measure thatthe mapping must stay under in order to be used. This threshold can beset to a value that is sufficient for high precision mapping (e.g.,registration), but such that mappings can be generated with highfrequency (i.e., the speed of the generation of the mapping does notrate limit a surgical procedure).

Prior to registration, there is no defined relationship between thecoordinate system that defines the anatomical model of the patient inthe medical imaging data and the coordinate system that defines thelocation of a surgical instrument attached to the robotic arm of therobotic surgical system. By generating a coordinate mapping from therobot coordinate system to the medical image data coordinate system andstoring the coordinate mapping, a processor can determine a physicallocation for each point of the patient’s anatomy represented in theanatomical model. In this way, the patient is registered with therobotic surgical system.

Once a patient is registered, for each point in space, the roboticsurgical system knows whether that point is on the surface of thepatient’s anatomy, in the patient’s anatomy, or outside of the patient’sanatomy. This information can be used for further processing to assistin surgical guidance and navigation. For example, this information canbe used to define “no go” zones for a surgical instrument if thepatient’s anatomy is to be avoided entirely or only a portion of thepatient’s anatomy is to be accessible to the surgical instrument.Additionally, a surgical instrument could trace a line, plane, or curvethat falls on the surface of the patient’s anatomy.

In order to accurately register a patient using haptic-feedback-basedcontacting, as described above, only a small set of points need to becontacted. For example, in certain embodiments, no more than 30 pointsare needed to register a patient’s anatomy to a robotic surgical systemwith sufficient precision to proceed with surgery. In certainembodiments, only 5-10 contacts are needed. The number of contactsnecessary varies with the particular surgical procedure being performed.In general, surgeries requiring more precise surgical instrumentpositioning require more contacts to be made in order to generate alarger set of spatial coordinates. Given the simplicity of using arobotic surgical system to contact the patient’s anatomy, a sufficientnumber of contacts for registration may be made in a short period oftime, thus expediting the overall surgical procedure.

In certain embodiments, the coordinate mapping is used to generatenavigational renderings on a display, for example, where a terminalpoint of a surgical instrument is shown in correct relation to thepatient’s anatomy. This rendering can be live updated as the position ofthe surgical tool shifts. This is done without the need for navigationalmarkers because the location of the surgical tool is known fromregistration. In certain embodiments, the set of spatial coordinates iscollected by contacting a fiducial marker with a plurality of orientingpoints (e.g., indents) on the marker (e.g., distributed on faces of themarker}, wherein the orienting points are in a known location relativeto the patient’s anatomy due to the marker being engineered to attach ina specific location on the patient’s anatomy (see FIG. 9 ). In certainembodiments, the specific location is the spinous process of a vertebra.

In some embodiments of methods and systems described herein,registration is performed using a technique known in the art.

During certain surgical procedures, a portion of a patient’s anatomy(i.e., a first volume) originally included in a model of the patient’sanatomy is removed during an operation prior to the removal of anadditional volume, for example, to gain access to the additional volume.The removal of the first volume may not require high precision duringremoval. The removal of the first volume may be done manually. The modelof the patient’s anatomy can be updated to reflect this removal. Themodel update may be necessary to maintain accurate patient records(i.e., medical history). The model update may be necessary forintra-operative planning of additional volume removal.

A set of spatial coordinates can be used to update the model of apatient’s anatomy after volume removal. In certain embodiments, the setof spatial coordinates are generated using haptic-feedback-hazedcontacting of the patient’s anatomy with an instrument attached to arobotic arm.

Using a set of spatial coordinates and a model of the patient’s anatomy,points on the patient’s anatomy that were formerly inside the surface ofthe anatomy can be determined, if a coordinate mapping between themodel’s coordinate system and the coordinate system of the spatialcoordinates (i.e., a robot’s coordinate system) has been generated. Thecoordinate mapping may be generated, for example, during registration.The set of spatial coordinates can be converted to be expressed in thecoordinate system of the model using the coordinate mapping. Then,coordinates determined to be located on the interior of the model’ssurface can be used to define a new surface for the model. For example,if half of a rectangular solid is removed, an instrument attached to arobotic surgical system can contact points that were previously on theinside of the rectangular solid, thus, the coordinates of the pointswhen converted to the model’s coordinate system will be located insidethe model. These points can be used to determine a new surface for themodel. Thus, the volume of the model will shrink by excluding all pointsin the removed volume from the model. The updated model will have asmaller volume than the original model that accurately reflects thechange in size that occurred due to volume removal. A coordinate may beincluded in a set of coordinates that is not determined to be aninternal coordinate in the model (i.e., the model before updating). Thiscoordinate would not be used to define a new surface as it would be partof an existing surface. This coordinate is thus not used to update themodel.

The revised set of coordinates that define the new surface of thepatient’s anatomy after volume removal can be stored as an updated modelfor future reference. This updated model may be stored in addition to oroverwrite the original model. When patient data is augmented to compriseboth the original model before volume removal and the updated modelafter volume removal, a comparison can be displayed. This comparisonincludes displaying the original model, the updated model, and theportion that was removed. An exemplary original model is shown in FIG.16 . An exemplary model highlighting the removed portion is shown inFIG. 17 . An exemplary updated model after volume removal is shown inFIG. 28 .

Because typical registration procedures are lengthy and requireunobstructed line- of-sight (either visually or electromagneticallyunobstructed line-of-sight), a navigation system and/or robotic surgicalsystem are typically only registered once during a surgical procedure,at the beginning. However, a patient’s orientation and/or positionrelative to these systems may shift during the procedure. Additionally,depending on the type of procedure, the patient’s anatomy may undergophysical changes that should be reflected in the registration. Seriousmedical error can result during a surgical procedure due to anydesynchronization that occurs between physical reality and the initialregistration. In general, more complex procedures involving many stepsare more prone to desynchronization and with greater magnitude. Seriousmedical error is more likely in surgical procedures on or near sensitiveanatomical features (e.g., nerves or arteries) due to desynchronization.Thus, easy and fast re-registration that may be performedintra-operatively is of great benefit.

Re-registration acts to reset any desynchronization that may havehappened and, unlike traditional registration methods,haptic-feedback-based contacting with robotic surgical systems are easyto integrate methods for re-registration. In certain embodiments, a re-registration can be processed in 1 - 2 seconds after re-registrationcontacts are made. In certain embodiments, re- registration can beprocessed in under 5 seconds. Re-registration may be performed with sucha system or using such a method by contacting the patient’s anatomy atany plurality of points. There is no need to contact the patient’sanatomy at specific points, for example, the points contacted duringinitial registration.

After an initial registration is performed that defines a coordinate mapbetween a robot’s coordinate system and the coordinate system of a modelof the patient’s anatomy, re-registration may be performed. Eachcoordinate of the patient’s anatomy is known to the robotic surgicalsystem after registration by expressing coordinates of the patient’sanatomical model in the robot’s coordinate system using a coordinatemapping. By collecting a set of spatial coordinates, the surface of thepatient’s anatomy expressed in the robot’s coordinate system can bemapped to the surface defamed by the set of spatial coordinates. Themapping can be used to update the coordinate mapping.

An updated coordinate mapping may reflect changes in the patient’sanatomy, orientation, or position. For example, if a patient is rotatedabout an axis, the patient’s physical anatomy will be tilted relative towhat the patient’s anatomical model reflects when expressed in a robot’scoordinate system. An instrument can contact the patient’s anatomy afterthe rotation at a set of points on the patient’s anatomy (for example,5-10 points). A re-registration map between the current patient’sanatomical model expressed in the robot’s coordinate system and thesurface defamed by the set of points is generated. The coordinate mapcan be modified using the registration map to produce an updatedcoordinate map. For example, if both the coordinate map andre-registration map are linear transforms stored as arrays, the updatedcoordinate map is the product of the two arrays. Likewise, a change inposition will be reflected as a translation during re-registration and achange in the patient’s anatomy may be reflected as a scaling transform.For example, the spacing of a patient’s vertebrae may change aftervolume removal, prompting a surgeon to perform re-registration.

In certain embodiments, the model of the patient’s anatomy is updatedsimultaneously during re-registration if one or more of the set ofspatial coordinates is determined to be an interior coordinate of themodel (i.e., the model as it existed pre- re- registration). Forexample, in certain embodiments, re-registration may be performed aftera volume removal whereby re-registration and model updating areprocessed in one simultaneous method. Re-registration is useful aftervolume removal because given the likelihood of anatomical shiftingduring such a significant surgical step is high.

In certain surgical procedures, the use of a surgical instrument shouldbe constrained to only a specific operational volume corresponding tothe surgical site of the patient’s anatomy. Operational volumes arephysical volumes, wherein the physical volume is defined using therobot’s coordinate system. It is clear that the physical volume in whichthe surgical instrument should be constrained is relative to thepatient’s anatomy. In certain embodiments, a robotic surgical systemprovides haptic feedback when a surgeon attempts to move the surgicalinstrument outside of the operational volume. In this way, the surgeonfeels resistance when the surgical instrument is at the boundary of theoperational volume and can redirect the surgical tool away from theboundary. This is useful, for example, during bone removal, where asurgeon does not want to remove bone from locations outside of theintended volume. Operational volumes are stored on a non-transitorycomputer readable medium for use in providing haptic feedback tosurgeons during surgical procedures.

Using a coordinate mapping, a physical operational volume occupied by avolume of and/or around a patient’s anatomy can be precisely definedusing the patient’s anatomical model. The intended operational volumecan be defined intra-operatively. A surgeon can contact a set of pointson the patient’s anatomy to generate a set of spatial coordinates thatdefine a boundary of the operational volume. In many surgicalprocedures, the operational volume corresponds to an anatomical featureof the patient’s anatomy. For example, a particular bone or segment of abone. Thus, in certain embodiments, the surgeon selects the anatomicalfeature from a model of the patient’s anatomy. The model of thepatient’s anatomical feature can be mapped to the set of spatialcoordinates corresponding to the surgeon’s desired operational volumeboundary. In certain embodiments, each coordinate in the model of thepatient’s anatomical feature can expressed using the robot’s coordinatesystem based on the mapping.

Then, the operational volume can be defined and expressed in the robot’scoordinate system using the mapping of the model of the patient’sanatomical feature to the set of spatial coordinates. In someembodiments, each coordinate of the surface of the model of thepatient’s anatomical feature is used to define the surface of theoperational volume.

In certain embodiments, the operational volume is the volume defined bythe surface defined by the set of spatial coordinates. Thus, thesemethods for producing operational volumes do not require medical imagingdata or a pre-constructed model of the patient’s anatomy. For example, asurgeon may contact 10 points on a patient that are determined as a setof spatial coordinates. The set of spatial coordinates can be used as aset of vertices (i.e., surface points) that define a volume. This may beused as an operational volume wherein movement of a surgical instrumentattached to the robotic arm of the robotic surgical system isconstrained to that operational volume. This can be done with or withoutany registration.

An operational volume may be defined by a surgeon selecting a volume ofa patient’s anatomical model without generating a set of spatialcoordinates (i.e., without contacting the patient’s anatomy). Thesurgeon may select the volume using software that allows viewing of thepatient’s anatomical model. Once the volume is selected, a coordinatemapping that maps the anatomical model to the robot’s coordinate systemmay be used to generate a set of coordinates that define the operationalvolume. Thus, the operational volume is defined using a coordinatemapping generated during, for example, registration or re-registration,and does not require a surgeon to separately generate a set of spatialcoordinates for use in defining the operational volume by contacting thepatient’s anatomy.

Stored operational volumes may be updated when a coordinate mapping isupdated or at some later time using the updated coordinate mapping. Thecoordinate mapping is updated during re- registration, for example, toreflect a shift in the position, orientation, or change in the patient’sanatomy. The coordinates of the stored operational volume may beconverted to the new robot coordinate system by mapping the new robotcoordinate system to the robot coordinate system the stored operationalvolume is expressed in and using the mapping (i.e., by converting eachcoordinate of the operational volume to the new coordinate system andstoring the converted coordinates as the updated operational volume).

Similarly, any stored model volume that is expressed in a medical imagedata coordinate system can be converted to a spatial volume byexpressing the coordinates of the model volume in the robot’s coordinatesystem using a coordinate mapping. This allows a robotic surgical systemto trace, enter, maneuver only within or maneuver only outside aphysical volume corresponding to the model volume.

A surgeon benefits from visualizing the position of a surgicalinstrument relative to a patient’s anatomy to assist in navigation anddecision making during a surgical procedure. When the terminal point ofa surgical instrument has a pre-defined position relative to the originof a robot’s coordinate system, the terminal point can be converted to aposition in a medical image data coordinate system with a coordinatemapping between the robot’s coordinate system and the medical image datacoordinate system using the position of the robotic arm. In certainembodiments, the terminal point is used to represent the position of therobotic arm. In certain embodiments, all surgical instruments have thesame terminal point when attached to a robotic arm. After converting theterminal point to be expressed in a medical image data coordinatesystem, the terminal point can be plotted relative to a model of thepatient’s anatomy that precisely reflects the distance between theterminal point and the patient’s anatomy in physical space. Using this,rendering data can be generated that, when displayed, shows arepresentation of the terminal point (and, optionally, the surgicalinstrument) and the model of the patient’s anatomy. Thus, visualizationof the surgical instrument and its position relative to the patient’sanatomy can be made without the use of a navigational marker attached tothe surgical instrument and without using image recognition techniques.

The rendering data may be displayed on a screen that is part of therobotic surgical system or that is viewable from inside and/or outsidean operating room. The screen may be mounted on the robotic arm. Therendering data may be updated to refresh the display in real-time orsubstantially real-time (i.e., acting as a video feed of the patient).The representation of the terminal point and/or surgical instrument maybe overlaid over a representation of the model of the patient’s anatomyor over medical images taken pre- or intra-operatively. The position ofthe terminal point on a display will update as the position of therobotic arm is adjusted.

FIG. 9 shows an exemplary fiducial marker that can be used inregistration and/or re-registration. The fiducial marker has indentsdistributed across different faces of an orientation member. Here, theorientation member is a cube. The attachment member of the fiducialmarker is attached to the spinous process of a vertebra, but can equallybe adapted to attach to any desired point on patient’s anatomy. When thesurgeon contacts the orientation points (i.e., indents) using ahaptic-feedback-contacting method as described herein, a coordinatemapping can be generated if the orientation points have a definedspatial relationship to the patient anatomy.

The following is a description of an exemplary surgical method forperforming a laminectomy, but it is understood that the method caneasily be adapted to other types of volume removal surgery.Additionally, many of the steps discussed in this exemplary method areapplicable to surgical procedures that do not involve volume removal orinvolve surgical outcomes other than volume removal. Aspects of thesurgical method are shown in FIGS. 14-35 .

FIG. 14 shows an exemplary robotic surgical system that can be used toperform a laminectomy using robotic-based navigation. FIG. 15 shows anexemplary navigation display that can be used by a surgeon forpre-operative planning, for example, to identify an operational volumeof the patient’s anatomy using the model of the patient’s anatomydisplayed on the display. FIG. 16 shows an additional exemplarynavigational display where the segmented vertebra will be the site ofthe laminectomy. Using preoperative planning, an initial volante to beremoved has been identified in the model of the patient’s anatomy. Theinitial volante is identified in light blue in FIG. 17 . FIG. 18 shows asurgeon using the robotic surgical system to receive real-time feedback.The real-time feedback comprises haptic feedback from the surgicalinstrwnent attached to the robotic surgical system and visual feedbackon the navigation screen attached to the robotic arm.

FIGS. 19-24 show a surgeon registering the patient to the roboticsurgical system. FIGS. 19- 22 show views of a navigation display whilethe patient is registered. The navigation shows two perspectives ofmedical image data that models the patient’s anatomy and arepresentation of the surgical instrwnent. In these figures, thesurgical instrwnent is touching points of the patient’s anatomy (i.e.,the patient’s vertebra). FIGS. 23 and 24 show the surgical instrwnentand robotic arm in various stages of contacting the patient’s anatomy togenerate a set of spatial coordinates during registration.

FIG. 25 shows a surgeon manually removing the spinous process of thevertebra. A volume removal can be performed manually or with a surgicalinstrwnent. Additionally, a volume removal can be performed freely,guided only by the surgeon (i.e., without defining an operationalvolume). In the case of laminectomy procedures, the spinous process isnot important and is not and is not near a sensitive part of thepatient’s anatomy, so manual removal is sufficient. FIGS. 26 - 27 show asurgeon in the process of updating the model of the patient’s anatomy bycontacting the patient’s anatomy in a plurality of points to generate aset of spatial coordinates. Because the set of spatial coordinatesgenerated in the process shown in FIGS. 26 - 27 will include spatialcoordinates for points that were previously interior points of thepatient’s anatomy, the patient’s anatomical model can be updatedaccordingly. FIG. 28 shows the updated anatomical model with region ofthe model near the new surface highlighted in medium blue. FIG. 29 showsa surgeon in the process of re-registering the patient. The surgeon cancontact the patient’s anatomy at any point. There is no need to contactthe patient at a specific point in order for successful re-registrationto occur. In the case of the exemplary method, the re- registration willbe performed based on the model of the patient updated to reflectremoval of the spinous process. FIGS. 30 and 31 show a navigationaldisplay viewed by the surgeon during the re- registration process. Thesurgeon can see the surgical instrument contacting the bone of thepatient to provide an additional check that the registration isaccurate. For example, if the surgeon was physically contacting thepatient’s anatomy with the surgical instrument but the navigationdisplay was not showing the terminal point of the surgical instrument atthe surface of the patient’s anatomy, then the surgeon would know thatre- registration was necessary before proceeding with the surgicalmethod.

FIG. 32 shows the surgeon maneuvering the surgical instrument within anoperational volume. The surgical instrument can be used to remove volumewithin the operational volume without risk to the surrounding volumes.During a laminectomy, the surgeon will operate close to a spinal nerve,so precision is critical to surgical outcomes. If the surgeon reachesthe boundary of the operational volume, the robotic arm will beprevented from moving further and haptic feedback will signal to thesurgeon to redirect movement away from and interior to the boundary.FIG. 33 shows a navigation display the surgeon uses during volumeremoval in the exemplary method. The surgeon can see the representationof the surgical instrument and its terminal point, medical image datamodeling the patient’s anatomy, and the operational volume of thecurrent procedure. The surgeon can use this to intra-operativelyredefine the operational volume if the original operational volume isdetermined to be insufficient or defective in some way. The overlay ofthe terminal point and the operational volume also provides a visualcheck for the surgeon that the registration is accurate. If the surgeonwas feeling haptic feedback, but the terminal point appeared in theoperational volume, the operational volume needs to be redefined and/orthe patient needs to be re-registered.

FIGS. 34 and 35 show a patient’s vertebra after completion of thelaminectomy. The necessary volume was removed. By using an operationalvolume during the volume removal in combination with the use of therobotic surgical system, the volume removal can take place significantlyfaster than equivalent volume removals with known techniques. Forexample, in certain embodiments, the volume removal of a laminectomyoccurs in 5 minutes or less, as compared to previously known techniquesthat generally take 30 minutes or more.

It is understood that surgical methods described herein are exemplary.Many surgical procedures require well-defined spatial relationshipsbetween a patient’s anatomy and surgical instruments as well asoperative volumes that constrain the movement of the surgicalinstruments. Other orthopedic and non-orthopedic methods are easilyadapted to integrate the methods described herein. Other surgeriescontemplated for use with robotic-based navigation systems and methodsinclude, but are not limited to, orthopedic procedures, ENT procedures,and neurosurgical procedures. It is readily understood by one ofordinary skill in the art that such procedures may be performed using anopen, percutaneous, or minimally invasive surgical (MIS) approach. It isalso understood that any of the methods for determining coordinatesand/or volumes using relevant coordinate systems described herein abovecan be used in any surgical method described herein.

FIG. 36 shows an illustrative network environment 3600 for use in themethods and systems described herein. In brief overview, referring nowto FIG. 36 , a block diagram of an exemplary cloud computing environment3600 is shown and described. The cloud computing environment 3600 mayinclude one or more resource providers 3602 a, 3602 b, 3602 c(collectively, 3602). Each resource provider 3602 may include computingresources. In some implementations, computing resources may include anyhardware and/or software used to process data. For example, computingresources may include hardware and/or software capable of executingalgorithms, computer programs, and/or computer applications. In someimplementations, exemplary computing resources may include applicationservers and/or databases with storage and retrieval capabilities. Eachresource provider 3602 may be connected to any other resource provider3602 in the cloud computing environment 3600. In some implementations,the resource providers 3602 may be connected over a computer network3608. Each resource provider 3602 may be connected to one or morecomputing device 3604 a, 3604 b, 3604 c (collectively, 3604), over thecomputer network 3608.

The cloud computing environment 3600 may include a resource manager3606. The resource manager 3606 may be connected to the resourceproviders 3602 and the computing devices 3604 over the computer network3608. In some implementations, the resource manager 3606 may facilitatethe provision of computing resources by one or more resource providers3602 to one or more computing devices 3604. The resource manager 3606may receive a request for a computing resource from a particularcomputing device 3604. The resource manager 3606 may identify one ormore resource providers 3602 capable of providing the computing resourcerequested by the computing device 3604. The resource manager 3606 mayselect a resource provider 3602 to provide the computing resource. Theresource manager 3606 may facilitate a connection between the resourceprovider 3602 and a particular computing device 3604. In someimplementations, the resource manager 3606 may establish a connectionbetween a particular resource provider 3602 and a particular computingdevice 3604. In some implementations, the resource manager 3606 mayredirect a particular computing device 3604 to a particular resourceprovider 3602 with the requested computing resource.

FIG. 37 shows an example of a computing device 3700 and a mobilecomputing device 750 that can be used in the methods and systemsdescribed in this disclosure. The computing device 3700 is intended torepresent various forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers. The mobile computing device3750 is intended to represent various forms of mobile devices, such aspersonal digital assistants, cellular telephones, smart- phones, andother similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexamples only, and are not meant to be limiting.

The computing device 3700 includes a processor 3702, a memory 3704, astorage device 3706, a high-speed interface 3708 connecting to thememory 3704 and multiple high-speed expansion ports 3710, and alow-speed interface 3712 connecting to a low-speed expansion port 3714and the storage device 3706. Each of the processor 3702, the memory3704, the storage device 3706, the high-speed interface 3708, thehigh-speed expansion ports 3710, and the low-speed interface 3712, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 3702 canprocess instructions for execution within the computing device 3700,including instructions stored in the memory 3704 or on the storagedevice 3706 to display graphical information for a GUI on an externalinput/output device, such as a display 3716 coupled to the high-speedinterface 3708. In other implementations, multiple processors and/ormultiple buses may be used, as appropriate, along with multiple memoriesand types of memory. Also, multiple computing devices may be connected,with each device providing portions of the necessary operations (e.g.,as a server bank, a group of blade servers, or a multi-processorsystem).

The memory 3704 stores information within the computing device 3700. Insome implementations, the memory 3704 is a volatile memory unit orunits. In some implementations, the memory 3704 is a non-volatile memoryunit or units. The memory 3704 may also be another form ofcomputer-readable medium, such as a magnetic or optical disk.

The storage device 3706 is capable of providing mass storage for thecomputing device 3700. In some implementations, the storage device 3706may be or contain a computer- readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. Instructions can be stored in an information carrier.The instructions, when executed by one or more processing devices (forexample, processor 3702), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices such as computer- or machine- readable mediums (forexample, the memory 3704, the storage device 3706, or memory on theprocessor 3702).

The high-speed interface 3708 manages bandwidth-intensive operations forthe computing device 3700, while the low-speed interface 3712 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In some implementations, the high- speed interface 3708 iscoupled to the memory 3704, the display 3716 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 3710,which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 3712 is coupled to the storagedevice 3706 and the low-speed expansion port 3714. The low-speedexpansion port 3714, which may include various communication ports(e.g., USB, Bluetooth®, Ethernet, wireless Ethernet) may be coupled toone or more input/output devices, such as a keyboard, a pointing device,a scanner, or a networking device such as a switch or router, e.g.,through a network adapter.

The computing device 3700 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 3720, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 3722. It may also be implemented as part of a rack serversystem 3724. Alternatively, components from the computing device 3700may be combined with other components in a mobile device (not shown),such as a mobile computing device 3750. Each of such devices may containone or more of the computing device 3700 and the mobile computing device3750, and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 3750 includes a processor 3752, a memory3764, an input/output device such as a display 3754, a communicationinterface 3766, and a transceiver 3768, among other components. Themobile computing device 3750 may also be provided with a storage device,such as a micro-drive or other device, to provide additional storage.Each of the processor 3752, the memory 3764, the display 3754, thecommunication interface 3766, and the transceiver 3768, areinterconnected using various buses, and several of the components may bemounted on a common motherboard or in other manners as appropriate.

The processor 3752 can execute instructions within the mobile computingdevice 3750, including instructions stored in the memory 3764. Theprocessor 3752 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 3752may provide, for example, for coordination of the other components ofthe mobile computing device 3750, such as control of user interfaces,applications run by the mobile computing device 3750, and wirelesscommunication by the mobile computing device 3750.

The processor 3752 may communicate with a user through a controlinterface 3758 and a display interface 3756 coupled to the display 3754.The display 3754 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface3756 may comprise appropriate circuitry for driving the display 3754 topresent graphical and other information to a user. The control interface3758 may receive commands from a user and convert them for submission tothe processor 3752. In addition, an external interface 3762 may providecommunication with the processor 3752, so as to enable near areacommunication of the mobile computing device 3750 with other devices.The external interface 3762 may provide, for example, for wiredcommunication in some implementations, or for wireless communication inother implementations, and multiple interfaces may also be used.

The memory 3764 stores information within the mobile computing device3750. The memory 3764 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 3774 may also beprovided and connected to the mobile computing device 3750 through anexpansion interface 3772, which may include, for example, a SIMM (Singlein Line Memory Module) card interface. The expansion memory 3774 mayprovide extra storage space for the mobile computing device 3750, or mayalso store applications or other information for the mobile computingdevice 3750. Specifically, the expansion memory 3774 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 3774 may be provided as a security module for themobile computing device 3750, and may be programmed with instructionsthat permit secure use of the mobile computing device 3750. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. In someimplementations, instructions are stored in an information carrier and,when executed by one or more processing devices (for example, processor3752), perform one or more methods, such as those described above. Theinstructions can also be stored by one or more storage devices, such asone or more computer- or machine-readable mediums (for example, thememory 3764, the expansion memory 3774, or memory on the processor3752). In some implementations, the instructions can be received in apropagated signal, for example, over the transceiver 3768 or theexternal interface 3762.

The mobile computing device 3750 may communicate wirelessly through thecommunication interface 3766, which may include digital signalprocessing circuitry where necessary. The communication interface 3766may provide for communications under various modes or protocols, such asGSM voice calls (Global System for Mobile communications), SMS (ShortMessage Service), EMS (Enhanced Messaging Service), or MMS messaging(Multimedia Messaging Service), CDMA (code division multiple access),TOMA (time division multiple access), PDC (Personal Digital Cellular),WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS(General Packet Radio Service), among others. Such communication mayoccur, for example, through the transceiver 3768 using aradio-frequency. In addition, short-range communication may occur, suchas using a Bluetooth®, Wi-Fi™, or other such transceiver (not shown). Inaddition, a GPS (Global Positioning System) receiver module 3770 mayprovide additional navigation- and location-related wireless data to themobile computing device 3750, which may be used as appropriate byapplications running on the mobile computing device 3750.

The mobile computing device 3750 may also communicate audibly using anaudio codec 3760, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 3760 maylikewise generate audible sound for a user, such as through a speaker,e.g., in a handset of the mobile computing device 3750. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 3750.

The mobile computing device 3750 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 3780. It may also be implemented aspart of a smart-phone 3782, personal digital assistant, or other similarmobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Certain embodiments of the present invention were described above. It is,however, expressly noted that the present invention is not limited tothose embodiments, but rather the intention is that additions andmodifications to what was expressly described herein are also includedwithin the scope of the invention. Moreover, it is to be understood thatthe features of the various embodiments described herein were notmutually exclusive and can exist in various combinations andpermutations, even if such combinations or permutations were not madeexpress herein, without departing from the spirit and scope of theinvention. In fact, variations, modifications, and other implementationsof what was described herein will occur to those of ordinary skill inthe art without departing from the spirit and the scope of theinvention. As such, the invention is not to be defined only by thepreceding illustrative description.

Having described certain implementations of methods and apparatus forrobotic navigation of robotic surgical systems, it will now becomeapparent to one of skill in the art that other implementationsincorporating the concepts of the disclosure may be used. Therefore, thedisclosure should not be limited to certain implementations, but rathershould be limited only by the spirit and scope of the following claims.

What is claimed is:
 1. A method for registering a patient’s anatomy withrobotic surgical system having an instrument attached to an end-effectorof a robotic arm of the robotic surgical system, the method comprisingthe steps of: providing the robotic arm including an end-effector havingan instrument attached thereto; providing a force sensor attacheddirectly or indirectly to the robotic arm.; and providing a processorand a memory having instructions stored thereon, wherein theinstructions, when executed by the processor, cause the processor to:receive haptic feedback, from the force sensor, prompted by movement ofthe instrument attached to the end-effector; determine that the hapticfeedback corresponds to contact of the instrument with a bone; determinea set of spatial coordinates, wherein the set of spatial coordinatescomprises a spatial coordinate for each contact of the instrument withthe bone, expressed using a robot coordinate system, wherein eachspatial coordinate corresponds to a point on the surface of ananatomical volume; determine a set of medical image data coordinatesexpressed using a medical image data coordinate system that correspondto a patient anatomy surface; map the surface corresponding to the setof spatial coordinates to the patient anatomy surface corresponding tothe set of medical image data; generate a coordinate mapping between therobot coordinate system and the medical image data coordinate systembased on the mapping between the surface corresponding to the set ofspatial coordinates and the surface corresponding to the set of medicalimage data coordinates; modifying, by the processor, the set of medicalimage data coordinates that define the surface of the volume of thepatient’s anatomy with a set of interior medical image data coordinatessuch that a first volume defined by the set of medical image datacoordinates is larger than a second volume defined by the modified setof medical image data coordinates; and storing the coordinate mapping,thereby registering the patient’s anatomy based on the determination ofcontact of the instrument with the bone.
 2. The method of claim 1,wherein the instructions, when executed by the processor, cause theprocessor to: output rendering data for display, wherein the renderingdata corresponds to a representation of a position of a member and atleast a portion of the medical image data based on the coordinatemapping, wherein the member is selected from the group consisting of:the end- effector, the instrument, and a surgical instrument.
 3. Themethod of claim 2, wherein the instructions, when executed by theprocessor, cause the processor to: generate new rendering data bymodifying the rendering data based on a change in the end-effector’sposition; and output the new rendering data for display.
 4. The methodof claim 1, wherein the end-effector includes a fiducial marker withknown size and shape such that the spatial coordinate is determinedusing a spatial relationship between the fiducial marker and thepatient’s anatomy.
 5. A method for registering a patient’s anatomy witha robotic surgical system having an instrument attached to anend-effector of a robotic arm of the robotic surgical system, the methodcomprising the steps of: providing a force sensor attached to theinstrument; and providing a processor and a memory having instructionsstored thereon, wherein the instructions, when executed by theprocessor, cause the processor to: receive haptic feedback, from theforce sensor, prompted by movement of the instrument attached to theend-effector; determine that the haptic feedback corresponds to contactof the instrument with a bone; determine a set of spatial coordinates,wherein the set of spatial coordinates comprises a spatial coordinatefor each contact of the instrument with the material, expressed using arobot coordinate system, wherein each spatial coordinate corresponds toa point on the surface of an anatomical volume; determine a set ofmedical image data coordinates expressed using a medical image datacoordinate system that correspond to a patient anatomy surface; map thesurface corresponding to the set of spatial coordinates to the patientanatomy surface corresponding to the set of medical image data; generatea coordinate mapping between the robot coordinate system and the medicalimage data coordinate system based on the mapping between the surfacecorresponding to the set of spatial coordinates and the surfacecorresponding to the set of medical image data coordinates; modifying,by the processor, the set of medical image data coordinates that definethe surface of the volume of the patient’s anatomy with the set ofinterior medical image data coordinates such that a first volume definedby the set of medical image data coordinates is larger than a secondvolume defined by the modified set of medical image data coordinates;and store the coordinate mapping, thereby dynamically registering thepatient’s anatomy based on the determination of contact of theinstrument with the bone.